Apply for flink contributor permission

2018-11-20 Thread Zhu Zhu
Hi there, Could anyone kindly give me the contributor permission? My JIRA id is zhuzh. Thanks, Zhu

Re: Apply for flink contributor permission

2018-11-20 Thread Zhu Zhu
Thanks Till! Till Rohrmann 于2018年11月20日周二 下午6:17写道: > Welcome to the community Zhu. I've given you contributor permissions. > > Cheers, > Till > > On Tue, Nov 20, 2018 at 11:10 AM Zhu Zhu wrote: > > > Hi there, > > > > Could anyone kindly give me the

Re: [PROGRESS-UPDATE] Redesign Flink Scheduling, introducing dedicated Scheduler Component

2019-04-15 Thread Zhu Zhu
The new interface will be very helpful to extend the scheduling strategy. It also makes the scheduling and failover process much cleaner. Thanks Gary to bring up this proposal. I really like it. +1 for this proposal. Regards, Zhu zhijiang 于2019年4月15日周一 下午2:45写道: > Thanks for sharing the latest

Re: [DISCUSS] Backtracking for failover regions

2019-04-15 Thread Zhu Zhu
s first improvement proposal :-) > > [1] > > https://docs.google.com/document/d/1fstkML72YBO1tGD_dmG2rwvd9bklhRVauh4FSsDDwXU/edit?usp=sharing > > Cheers, > Till > > On Sun, Apr 14, 2019 at 8:20 PM Chesnay Schepler > wrote: > > > Hello everyone, >

Re: [DISCUSS] Allow at-most-once delivery in case of failures

2019-06-11 Thread Zhu Zhu
Thanks Xiaogang for initiating the discussion. I think it is a very good proposal. We also received this requirements for Flink from Alibaba internal and external customers. In these cases, users are less concerned of the data consistency, but have higher demands for low latency. Here are a

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

2019-08-14 Thread Zhu Zhu
Congratulations Andrey! Thanks, Zhu Zhu vino yang 于2019年8月15日周四 上午11:05写道: > Congratulations Andrey! > > Best, > Vino > > Yun Gao 于2019年8月15日周四 上午10:49写道: > > > Congratulations An

Re: [DISCUSS] Reducing build times

2019-08-15 Thread Zhu Zhu
it multiple times. With it we can postpone the testing of IT cases or connectors before the PR reaches a stable state. Thanks, Zhu Zhu Chesnay Schepler 于2019年8月15日周四 下午3:38写道: > Hello everyone, > > improving our build times is a hot topic at the moment so let's discuss > the differ

Re: How to load udf jars in flink program

2019-08-15 Thread Zhu Zhu
Hi Jiangang, Does "flink run -j jarpath ..." work for you? If that jar id deployed to the same path on each worker machine, you can try "flink run -C classpath ..." as well. Thanks, Zhu Zhu 刘建刚 于2019年8月15日周四 下午5:31写道: > We are using per-job to load udf jar when

Re: Checkpointing under backpressure

2019-08-14 Thread Zhu Zhu
nd I agree a slot with point 4 in Piotr's last mail and think it's necessary for a CP1 snapshotted task to process CP1 buffered data before all CP1 barriers are are received in this task. Otherwise it might be a processing performance regression compared to current exactly-once checkpointing. Thank

Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project

2019-07-22 Thread Zhu Zhu
Congratulations Zhijiang! boshu Zheng 于2019年7月23日周二 上午9:46写道: > Congrats Zhijiang :) > On 07/23/2019 09:29, Dian Fu wrote: > Congrats Zhijiang! > > > 在 2019年7月23日,上午9:14,Kurt Young 写道: > > > > Congratulations Zhijiang! > > > > Best, > > Kurt > > > > > > On Tue, Jul 23, 2019 at 8:59 AM Biao Liu

Re: [ANNOUNCE] Jiangjie (Becket) Qin has been added as a committer to the Flink project

2019-07-18 Thread Zhu Zhu
Congratulations Becket! Xintong Song 于2019年7月18日周四 下午4:33写道: > Congratulations Becket! > > Thank you~ > > Xintong Song > > > > On Thu, Jul 18, 2019 at 4:20 PM Kurt Young wrote: > > > Congrats Becket! > > > > Best, > > Kurt > > > > > > On Thu, Jul 18, 2019 at 4:12 PM JingsongLee > .invalid> >

Re: [DISCUSS] Allow at-most-once delivery in case of failures

2019-07-24 Thread Zhu Zhu
ver strategy. It can be helpful for scenarios with higher data consistency demands. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+Grained+Recovery+from+Task+Failures Thanks, Zhu Zhu Biao Liu 于2019年7月24日周三 上午10:41写道: > Hi Stephan & Xiaogang, > > It's great to

Re: [ANNOUNCE] Hequn becomes a Flink committer

2019-08-07 Thread Zhu Zhu
Congratulations to Hequn! Thanks, Zhu Zhu Zili Chen 于2019年8月7日周三 下午5:16写道: > Congrats Hequn! > > Best, > tison. > > > Jeff Zhang 于2019年8月7日周三 下午5:14写道: > >> Congrats Hequn! >> >> Paul Lam 于2019年8月7日周三 下午5:08写道: >> >>> Congrats Hequn! W

Re: CiBot Update

2019-08-22 Thread Zhu Zhu
Thanks Chesnay for the CI improvement! It is very helpful. Thanks, Zhu Zhu zhijiang 于2019年8月22日周四 下午4:18写道: > It is really very convenient now. Valuable work, Chesnay! > > Best, > Zhijiang > -- > From:Till Roh

Re: [DISCUSS] Use Java's Duration instead of Flink's Time

2019-08-24 Thread Zhu Zhu
+1 since Java Duration is more common and powerful than Flink Time. For whether to drop scala Duration for parsing duration OptionConfig, I think it's another question and should be discussed in another thread. Thanks, Zhu Zhu Becket Qin 于2019年8月24日周六 下午4:16写道: > +1, makes sense. BTW,

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

2019-08-24 Thread Zhu Zhu
Hi Piotr, Thanks for the explanation. Agreed that the broadcastEmit(record) is a better choice for broadcasting for the iterations. As broadcasting for the iterations is the first motivation, let's support it first. Thanks, Zhu Zhu Yun Gao 于2019年8月23日周五 下午11:56写道: > Hi Pi

Re: How to handle Flink Job with 400MB+ Uberjar with 800+ containers ?

2019-08-30 Thread Zhu Zhu
One optimization that we take is letting yarn to reuse the flink-dist jar which was localized when running previous jobs. Thanks, Zhu Zhu Jörn Franke 于2019年8月30日周五 下午4:02写道: > Increase replication factor and/or use HDFS cache > https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/

Re: [SURVEY] Is the default restart delay of 0s causing problems?

2019-08-30 Thread Zhu Zhu
In our production, we usually override the restart delay to be 10 s. We once encountered cases that external services are overwhelmed by reconnections from frequent restarted tasks. As a safer though not optimized option, a default delay larger than 0 s is better in my opinion. 未来阳光

Re: [DISCUSS] FLIP-53: Fine Grained Resource Management

2019-09-03 Thread Zhu Zhu
Thanks Xintong for the explanation. For question #1, I think it's good as long as DataSet job behaviors remains the same. For question #2, agreed that the resource difference is small enough(at most 1 edge diff) in current supported point-wise execution edge connection patterns. Thanks, Zhu Zhu

Re: [SURVEY] Is the default restart delay of 0s causing problems?

2019-09-02 Thread Zhu Zhu
1s looks good to me. And I think the conclusion that when a user should override the delay is worth to be documented. Thanks, Zhu Zhu Steven Wu 于2019年9月3日周二 上午4:42写道: > 1s sounds a good tradeoff to me. > > On Mon, Sep 2, 2019 at 1:30 PM Till Rohrmann wrote: > >> Thanks

Re: [DISCUSS] Simplify Flink's cluster level RestartStrategy configuration

2019-09-01 Thread Zhu Zhu
he default restart delay of 0s causing problems) is ongoing and we may need to take the result from that. Thanks, Zhu Zhu Becket Qin 于2019年9月2日周一 上午9:06写道: > +1. The new behavior makes sense to me. > > BTW, we need a FLIP for this :) > > On Fri, Aug 30, 2019 at 10:17 PM Till Rohrman

Re: How to handle Flink Job with 400MB+ Uberjar with 800+ containers ?

2019-09-01 Thread Zhu Zhu
ributed cache of YARN. In this way, the localized dist jar can be shared by different YARN applications and it will not be removed when the YARN application which localized it terminates. This requires some changes in Flink though. We will open a ISSUE to contribute this optimization to the community. Thank

Re: [DISCUSS] FLIP-53: Fine Grained Resource Management

2019-09-02 Thread Zhu Zhu
p 4 should be *StreamingJobGraphGenerator*, as *StreamGraphGenerator* is not aware of JobGraph and pipelined region. Thanks, Zhu Zhu Xintong Song 于2019年9月2日周一 上午11:59写道: > Updated the FLIP wiki page [1], with the following changes. > >- Remove the step of converting pipelined edges between differen

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

2019-08-23 Thread Zhu Zhu
, which is hard to maintain and extend. Thanks, Zhu Zhu Yun Gao 于2019年8月22日周四 下午8:42写道: > Hi everyone, > In some scenarios we met a requirement that some operators want to > send records to theirs downstream operators with an multicast communication > pattern. In detail, for

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

2019-08-23 Thread Zhu Zhu
ast can help with this case. Thanks, Zhu Zhu Piotr Nowojski 于2019年8月23日周五 下午3:20写道: > Hi, > > Yun: > > Thanks for proposing the idea. I have checked the document and left couple > of questions there, but it might be better to answer them here. > > What is the exact motivati

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

2019-08-23 Thread Zhu Zhu
Hi Piotr, Yes you are right it's a distributed cross join requirement. Broadcast join can help with cross join cases. But users cannot use it if the data set to join is too large to fit into one subtask. Sorry for left some details behind. Thanks, Zhu Zhu Piotr Nowojski 于2019年8月23日周五 下午4:57写道

Re: [DISCUSS] Enhance Support for Multicast Communication Pattern

2019-08-23 Thread Zhu Zhu
known scenario, I think users can benefit from cross join sooner or later. Thanks, Zhu Zhu Piotr Nowojski 于2019年8月23日周五 下午6:19写道: > Hi, > > Thanks for the answers :) Ok I understand the full picture now. +1 from my > side on solving this issue somehow. But before we start discussing ho

Re: [VOTE] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

2019-09-04 Thread Zhu Zhu
+1 (non-binding) Thanks, Zhu Zhu Till Rohrmann 于2019年9月4日周三 下午5:06写道: > Hi everyone, > > I would like to start the voting process for FLIP-62 [1], which > is discussed and reached consensus in this thread [2]. > > Since the change is rather small I'd like to shorten the vot

Re: [VOTE] FLIP-61 Simplify Flink's cluster level RestartStrategy configuration

2019-09-04 Thread Zhu Zhu
+1 (non-binding) Thanks, Zhu Zhu Till Rohrmann 于2019年9月4日周三 下午5:05写道: > Hi everyone, > > I would like to start the voting process for FLIP-61 [1], which is > discussed and reached consensus in this thread [2]. > > Since the change is rather small I'd like to shorten the vot

Re: [DISCUSS] Features for Apache Flink 1.10

2019-09-06 Thread Zhu Zhu
Thanks Gary for kicking off this discussion. Really appreciate that you and Yu offer to help to manage 1.10 release. +1 for Gary and Yu as release managers. Thanks, Zhu Zhu Dian Fu 于2019年9月7日周六 下午12:26写道: > Hi Gary, > > Thanks for kicking off the release schedule of 1.10. +1 for you

Re: [VOTE] FLIP-53: Fine Grained Operator Resource Management

2019-09-06 Thread Zhu Zhu
Thanks Xintong for proposing this better resource management. This helps a lot to users who want to better manage the job resources. And would be even more useful if in the future we can have auto-tuning mechanism for jobs. +1 (non-binding) Thanks, Zhu Zhu Xintong Song 于2019年9月6日周五 上午11:17写道

Re: Checkpointing clarification

2019-09-06 Thread Zhu Zhu
this state. Thanks, Zhu Zhu Dian Fu 于2019年9月6日周五 下午8:17写道: > When a WindowOperator receives all the barrier from the upstream, it will > forward the barrier to downstream operator and perform the checkpoint > asynchronously. > It doesn't have to wait the window to trigger before

Re: [ANNOUNCE] Kostas Kloudas joins the Flink PMC

2019-09-06 Thread Zhu Zhu
Congratulations Kostas! Thanks, Zhu Zhu Yu Li 于2019年9月6日周五 下午10:49写道: > Congratulations Klou! > > Best Regards, > Yu > > > On Fri, 6 Sep 2019 at 22:43, Forward Xu wrote: > > > Congratulations Kloudas! > > > > > > Best, > > > &

[SURVEY] How many people are using customized RestartStrategy(s)

2019-09-12 Thread Zhu Zhu
, Zhu Zhu

Re: [SURVEY] How many people are using customized RestartStrategy(s)

2019-09-12 Thread Zhu Zhu
estart-strategy: org.foobar.MyRestartStrategyFactoryFactory". The usage of restart strategies you mentioned will keep working with the new scheduler. Thanks, Zhu Zhu Oytun Tez 于2019年9月12日周四 下午10:05写道: > Hi Zhu, > > We are using custom restart strategy like this: > > environment.setRestartStrategy(f

Re: [ANNOUNCE] Zili Chen becomes a Flink committer

2019-09-11 Thread Zhu Zhu
Congratulations Zili! Thanks, Zhu Zhu Terry Wang 于2019年9月11日周三 下午5:34写道: > Congratulations! > > Best, > Terry Wang > > > > 在 2019年9月11日,下午5:28,Dian Fu 写道: > > Congratulations! > > 在 2019年9月11日,下午5:26,Jeff Zhang 写道: > > Congratulations Zili Che

[jira] [Created] (FLINK-8382) sheduleRunAsync with a positive schedule delay does not work in JobMaster

2018-01-05 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-8382: -- Summary: sheduleRunAsync with a positive schedule delay does not work in JobMaster Key: FLINK-8382 URL: https://issues.apache.org/jira/browse/FLINK-8382 Project: Flink

[jira] [Created] (FLINK-10240) Flexible scheduling strategy is needed for batch job

2018-08-29 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-10240: --- Summary: Flexible scheduling strategy is needed for batch job Key: FLINK-10240 URL: https://issues.apache.org/jira/browse/FLINK-10240 Project: Flink Issue Type: New

[jira] [Created] (FLINK-10412) toString field in AbstractID should be transient to avoid been serialized

2018-09-24 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-10412: --- Summary: toString field in AbstractID should be transient to avoid been serialized Key: FLINK-10412 URL: https://issues.apache.org/jira/browse/FLINK-10412 Project: Flink

[jira] [Created] (FLINK-10413) requestPartitionState messages overwhelms JM RPC main thread

2018-09-24 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-10413: --- Summary: requestPartitionState messages overwhelms JM RPC main thread Key: FLINK-10413 URL: https://issues.apache.org/jira/browse/FLINK-10413 Project: Flink Issue

[jira] [Created] (FLINK-11165) Refine the deploying log for easier finding of task locations

2018-12-14 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-11165: --- Summary: Refine the deploying log for easier finding of task locations Key: FLINK-11165 URL: https://issues.apache.org/jira/browse/FLINK-11165 Project: Flink Issue

[jira] [Created] (FLINK-10945) Avoid resource deadlocks for finite stream jobs when resources are limited

2018-11-20 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-10945: --- Summary: Avoid resource deadlocks for finite stream jobs when resources are limited Key: FLINK-10945 URL: https://issues.apache.org/jira/browse/FLINK-10945 Project: Flink

[jira] [Created] (FLINK-12131) Resetting ExecutionVertex in region failover may cause inconsistency of IntermediateResult status

2019-04-08 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-12131: --- Summary: Resetting ExecutionVertex in region failover may cause inconsistency of IntermediateResult status Key: FLINK-12131 URL: https://issues.apache.org/jira/browse/FLINK-12131

[jira] [Created] (FLINK-12138) Limit input split count of each source task for better failover experience

2019-04-09 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-12138: --- Summary: Limit input split count of each source task for better failover experience Key: FLINK-12138 URL: https://issues.apache.org/jira/browse/FLINK-12138 Project: Flink

[jira] [Created] (FLINK-12643) Implement ExecutionGraph to FailoverTopology Adapter

2019-05-28 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-12643: --- Summary: Implement ExecutionGraph to FailoverTopology Adapter Key: FLINK-12643 URL: https://issues.apache.org/jira/browse/FLINK-12643 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-12709) Implement RestartBackoffTimeStrategyFactoryLoader

2019-06-03 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-12709: --- Summary: Implement RestartBackoffTimeStrategyFactoryLoader Key: FLINK-12709 URL: https://issues.apache.org/jira/browse/FLINK-12709 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-12876) Adapt region failover NG for legacy scheduler

2019-06-17 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-12876: --- Summary: Adapt region failover NG for legacy scheduler Key: FLINK-12876 URL: https://issues.apache.org/jira/browse/FLINK-12876 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-12926) Main thread checking in some tests fails

2019-06-21 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-12926: --- Summary: Main thread checking in some tests fails Key: FLINK-12926 URL: https://issues.apache.org/jira/browse/FLINK-12926 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-12369) Introducing next version failover strategy interfaces and implement a region failover strategy with it

2019-04-29 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-12369: --- Summary: Introducing next version failover strategy interfaces and implement a region failover strategy with it Key: FLINK-12369 URL: https://issues.apache.org/jira/browse/FLINK-12369

[jira] [Created] (FLINK-13241) YarnResourceManager gives no response to second round slot allocations

2019-07-12 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-13241: --- Summary: YarnResourceManager gives no response to second round slot allocations Key: FLINK-13241 URL: https://issues.apache.org/jira/browse/FLINK-13241 Project: Flink

[jira] [Created] (FLINK-13254) Task launching blocked due to pending on #waitForChannel after failover

2019-07-14 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-13254: --- Summary: Task launching blocked due to pending on #waitForChannel after failover Key: FLINK-13254 URL: https://issues.apache.org/jira/browse/FLINK-13254 Project: Flink

[jira] [Created] (FLINK-13256) Periodical checkpointing is stopped after failovers

2019-07-15 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-13256: --- Summary: Periodical checkpointing is stopped after failovers Key: FLINK-13256 URL: https://issues.apache.org/jira/browse/FLINK-13256 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-13056) Optimize region failover performance on calculating vertices to restart

2019-07-02 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-13056: --- Summary: Optimize region failover performance on calculating vertices to restart Key: FLINK-13056 URL: https://issues.apache.org/jira/browse/FLINK-13056 Project: Flink

[jira] [Created] (FLINK-13055) Leverage JM side partition state to improve region failover experience

2019-07-02 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-13055: --- Summary: Leverage JM side partition state to improve region failover experience Key: FLINK-13055 URL: https://issues.apache.org/jira/browse/FLINK-13055 Project: Flink

[jira] [Created] (FLINK-13336) Remove the legacy batch fault tolerance page and redirect it to the new task failure recovery page

2019-07-19 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-13336: --- Summary: Remove the legacy batch fault tolerance page and redirect it to the new task failure recovery page Key: FLINK-13336 URL: https://issues.apache.org/jira/browse/FLINK-13336

[jira] [Created] (FLINK-13421) Unexpected ConcurrentModificationException when RM notify JM about allocation failure

2019-07-25 Thread Zhu Zhu (JIRA)
Zhu Zhu created FLINK-13421: --- Summary: Unexpected ConcurrentModificationException when RM notify JM about allocation failure Key: FLINK-13421 URL: https://issues.apache.org/jira/browse/FLINK-13421 Project

[jira] [Created] (FLINK-13887) ExecutionConfig#setDefaultInputDependencyConstraint should do NotNull check on params

2019-08-28 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-13887: --- Summary: ExecutionConfig#setDefaultInputDependencyConstraint should do NotNull check on params Key: FLINK-13887 URL: https://issues.apache.org/jira/browse/FLINK-13887 Project

[jira] [Created] (FLINK-13962) Execution#taskRestore leaks if task fails before deploying

2019-09-04 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-13962: --- Summary: Execution#taskRestore leaks if task fails before deploying Key: FLINK-13962 URL: https://issues.apache.org/jira/browse/FLINK-13962 Project: Flink Issue Type

[jira] [Created] (FLINK-14000) Remove legacy ProcessShutDownThread

2019-09-07 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-14000: --- Summary: Remove legacy ProcessShutDownThread Key: FLINK-14000 URL: https://issues.apache.org/jira/browse/FLINK-14000 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-14069) Enable TimeUtils to parse all time units labels supported by scala Duration

2019-09-12 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-14069: --- Summary: Enable TimeUtils to parse all time units labels supported by scala Duration Key: FLINK-14069 URL: https://issues.apache.org/jira/browse/FLINK-14069 Project: Flink

[jira] [Created] (FLINK-14070) Use TimeUtils to parse duration configs

2019-09-12 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-14070: --- Summary: Use TimeUtils to parse duration configs Key: FLINK-14070 URL: https://issues.apache.org/jira/browse/FLINK-14070 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-14040) Enable a cron job to run MiniCluster tests for schedulerNG

2019-09-10 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-14040: --- Summary: Enable a cron job to run MiniCluster tests for schedulerNG Key: FLINK-14040 URL: https://issues.apache.org/jira/browse/FLINK-14040 Project: Flink Issue Type