Thanks everyone for the input again. I'll then conclude this survey thread and start a discuss thread to set the default restart delay to 1s.
@Arvid, I agree that a better documentation how to tune Flink with sane settings for certain scenarios is super helpful. However, as you've said it is somewhat hijacking the discussion and I would exclude it from my proposed changes. The best thing to do would be to start a separate discussion/effort for it. Concerning the restart strategy configuration options, they are currently only documented here [1]. I'm about to change it with this PR [2]. [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/task_failure_recovery.html [2] https://github.com/apache/flink/pull/9562 Cheers, Till On Tue, Sep 3, 2019 at 8:21 AM Arvid Heise <ar...@data-artisans.com> wrote: > Hi all, > > just wanted to share my experience with configurations with you. For > non-expert users configurations of Flink can be very daunting. The list of > common properties is already helping a lot [1], but it's not clear how they > depend on each other and settings common for specific use cases are not > listed. > > If we can give somewhat clear recommendations for the start for the most > common use cases (batch small/large cluster, streaming high throughput/low > latency), I think users would be able start much more quickly with a > somewhat well-configured system and fine-tune the settings later. For > example, Kafka Streams has a section on how to set the parameters for > maximum resilience [2]. > > I'd propose to leave the current configuration page as a reference page, > but also have a recommended configuration settings page that's directly > linked in the first section, such that new users are not overwhelmed. > > Sorry if this response is hijacking the discussion. > Btw, is restart-strategy configuration missing in the main configuration > page? Is this a conscious decision? > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#common-options > [2] > https://docs.confluent.io/current/streams/developer-guide/config-streams.html#recommended-configuration-parameters-for-resiliency > > On Tue, Sep 3, 2019 at 5:10 AM Zhu Zhu <reed...@gmail.com> wrote: > >> 1s looks good to me. >> And I think the conclusion that when a user should override the delay is >> worth to be documented. >> >> Thanks, >> Zhu Zhu >> >> Steven Wu <stevenz...@gmail.com> 于2019年9月3日周二 上午4:42写道： >> >>> 1s sounds a good tradeoff to me. >>> >>> On Mon, Sep 2, 2019 at 1:30 PM Till Rohrmann <trohrm...@apache.org> >>> wrote: >>> >>>> Thanks a lot for all your feedback. I see there is a slight tendency >>>> towards having a non zero default delay so far. >>>> >>>> However, Yu has brought up some valid points. Maybe I can shed some >>>> light on a). >>>> >>>> Before FLINK-9158 we set the default delay to 10s because Flink did not >>>> support queued scheduling which meant that if one slot was missing/still >>>> being occupied, then Flink would fail right away with >>>> a NoResourceAvailableException. In order to prevent this we added the >>>> delay. This also covered the case when the job was failing because of an >>>> overloaded external system. >>>> >>>> When we finished FLIP-6, we thought that we could improve the user >>>> experience by decreasing the default delay to 0s because all Flink related >>>> problems (slot still occupied, slot missing because of reconnecting TM) >>>> could be handled by the default slot request time out which allowed the >>>> slots to become ready after the scheduling was kicked off. However, we did >>>> not properly take the case of overloaded external systems into account. >>>> >>>> For b) I agree that any default value should be properly documented. >>>> This was clearly an oversight when FLINK-9158 has been merged. Moreover, I >>>> believe that there won't be the solve it all default value. There are >>>> always cases where one needs to adapt it to ones needs. But this is ok. The >>>> goal should be to find the default value which works for most cases. >>>> >>>> So maybe the middle ground between 10s and 0s could be a solution. >>>> Setting the default restart delay to 1s should prevent restart storms >>>> caused by overloaded external systems and still be fast enough to not slow >>>> down recoveries noticeably in most cases. If one needs a super fast >>>> recovery, then one should set the delay value to 0s. If one requires a >>>> longer delay because of a particular infrastructure, then one needs to >>>> change the value too. What do you think? >>>> >>>> Cheers, >>>> Till >>>> >>>> On Sun, Sep 1, 2019 at 11:56 PM Yu Li <car...@gmail.com> wrote: >>>> >>>>> -1 on increasing the default delay to none zero, with below reasons: >>>>> >>>>> a) I could see some concerns about setting the delay to zero in the >>>>> very original JIRA (FLINK-2993 >>>>> <https://issues.apache.org/jira/browse/FLINK-2993>) but later on in >>>>> FLINK-9158 <https://issues.apache.org/jira/browse/FLINK-9158> we >>>>> still decided to make the change, so I'm wondering whether the decision >>>>> also came from any customer requirement? If so, how could we judge whether >>>>> one requirement override the other? >>>>> >>>>> b) There could be valid reasons for both default values depending on >>>>> different use cases, as well as relative work around (like based on latest >>>>> policy, setting the config manually to 10s could resolve the problem >>>>> mentioned), and from former replies to this thread we could see users have >>>>> already taken actions. Changing it back to non-zero again won't affect >>>>> such >>>>> users but might cause surprises to those depending on 0 as default. >>>>> >>>>> Last but not least, no matter what decision we make this time, I'd >>>>> suggest to make it final and document in our release note explicitly. >>>>> Checking the 1.5.0 release note [1] [2] it seems we didn't mention about >>>>> the change on default restart delay and we'd better learn from it this >>>>> time. Thanks. >>>>> >>>>> [1] >>>>> https://flink.apache.org/news/2018/05/25/release-1.5.0.html#release-notes >>>>> [2] >>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.5/release-notes/flink-1.5.html >>>>> >>>>> Best Regards, >>>>> Yu >>>>> >>>>> >>>>> On Sun, 1 Sep 2019 at 04:33, Steven Wu <stevenz...@gmail.com> wrote: >>>>> >>>>>> +1 on what Zhu Zhu said. >>>>>> >>>>>> We also override the default to 10 s. >>>>>> >>>>>> On Fri, Aug 30, 2019 at 8:58 PM Zhu Zhu <reed...@gmail.com> wrote: >>>>>> >>>>>>> In our production, we usually override the restart delay to be 10 s. >>>>>>> We once encountered cases that external services are overwhelmed by >>>>>>> reconnections from frequent restarted tasks. >>>>>>> As a safer though not optimized option, a default delay larger than >>>>>>> 0 s is better in my opinion. >>>>>>> >>>>>>> >>>>>>> 未来阳光 <2217232...@qq.com> 于2019年8月30日周五 下午10:23写道： >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> >>>>>>>> I thinks it's better to increase the default value. +1 >>>>>>>> >>>>>>>> >>>>>>>> Best. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ------------------ 原始邮件 ------------------ >>>>>>>> 发件人: "Till Rohrmann"<trohrm...@apache.org>; >>>>>>>> 发送时间: 2019年8月30日(星期五) 晚上10:07 >>>>>>>> 收件人: "dev"<d...@flink.apache.org>; "user"<user@flink.apache.org>; >>>>>>>> 主题: [SURVEY] Is the default restart delay of 0s causing problems? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi everyone, >>>>>>>> >>>>>>>> I wanted to reach out to you and ask whether decreasing the default >>>>>>>> delay >>>>>>>> to `0 s` for the fixed delay restart strategy [1] is causing >>>>>>>> trouble. A >>>>>>>> user reported that he would like to increase the default value >>>>>>>> because it >>>>>>>> can cause restart storms in case of systematic faults [2]. >>>>>>>> >>>>>>>> The downside of increasing the default delay would be a slightly >>>>>>>> increased >>>>>>>> restart time if this config option is not explicitly set. >>>>>>>> >>>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-9158 >>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-11218 >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Till >>>>>>> >>>>>>> > > -- > > Arvid Heise | Senior Software Engineer > > <https://www.ververica.com/> > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen >
