Hi Steven, As a conclusion, since we will have a meter metric[1] for restarts, customized restart strategy is not needed in your case. Is that right?
[1] https://issues.apache.org/jira/browse/FLINK-14164 Thanks, Zhu Zhu Steven Wu <stevenz...@gmail.com> 于2019年9月25日周三 上午2:30写道： > Zhu Zhu, > > Sorry, I was using different terminology. yes, Flink meter is what I was > talking about regarding "fullRestarts" for threshold based alerting. > > On Mon, Sep 23, 2019 at 7:46 PM Zhu Zhu <reed...@gmail.com> wrote: > >> Steven, >> >> In my mind, Flink counter only stores its accumulated count and reports >> that value. Are you using an external counter directly? >> Maybe Flink Meter/MeterView is what you need? It stores the count and >> calculates the rate. And it will report its "count" as well as "rate" to >> external metric services. >> >> The counter "task_failures" only works if the individual failover >> strategy is enabled. However, it is not a public interface and is not >> suggested to use, as the fine grained recovery (region failover) now >> supersedes it. >> I've opened a ticket[1] to add a metric to show failovers that respects >> fine grained recovery. >> >> [1] https://issues.apache.org/jira/browse/FLINK-14164 >> >> Thanks, >> Zhu Zhu >> >> Steven Wu <stevenz...@gmail.com> 于2019年9月24日周二 上午6:41写道： >> >>> >>> When we setup alert like "fullRestarts > 1" for some rolling window, we >>> want to use counter. if it is a Gauge, "fullRestarts" will never go below 1 >>> after a first full restart. So alert condition will always be true after >>> first job restart. If we can apply a derivative to the Gauge value, I guess >>> alert can probably work. I can explore if that is an option or not. >>> >>> Yeah. Understood that "fullRestart" won't increment when fine grained >>> recovery happened. I think "task_failures" counter already exists in Flink. >>> >>> >>> >>> On Sun, Sep 22, 2019 at 7:59 PM Zhu Zhu <reed...@gmail.com> wrote: >>> >>>> Steven, >>>> >>>> Thanks for the information. If we can determine this a common issue, we >>>> can solve it in Flink core. >>>> To get to that state, I have two questions which need your help: >>>> 1. Why is gauge not good for alerting? The metric "fullRestart" is a >>>> Gauge<Long>. Does the metric reporter you use report Counter and >>>> Gauge<Long> to external services in different ways? Or anything else can be >>>> different due to the metric type? >>>> 2. Is the "number of restarts" what you actually need, rather than >>>> the "fullRestart" count? If so, I believe we will have such a counter >>>> metric in 1.10, since the previous "fullRestart" metric value is not the >>>> number of restarts when grained recovery (feature added 1.9.0) is enabled. >>>> "fullRestart" reveals how many times entire job graph has been >>>> restarted. If grained recovery (feature added 1.9.0) is enabled, the graph >>>> would not be restarted when task failures happen and the "fullRestart" >>>> value will not increment in such cases. >>>> >>>> I'd appreciate if you can help with these questions and we can make >>>> better decisions for Flink. >>>> >>>> Thanks, >>>> Zhu Zhu >>>> >>>> Steven Wu <stevenz...@gmail.com> 于2019年9月22日周日 上午3:31写道： >>>> >>>>> Zhu Zhu, >>>>> >>>>> Flink fullRestart metric is a Gauge, which is not good for alerting >>>>> on. We publish an equivalent Counter metric for alerting purpose. >>>>> >>>>> Thanks, >>>>> Steven >>>>> >>>>> On Thu, Sep 19, 2019 at 7:45 PM Zhu Zhu <reed...@gmail.com> wrote: >>>>> >>>>>> Thanks Steven for the feedback! >>>>>> Could you share more information about the metrics you add in you >>>>>> customized restart strategy? >>>>>> >>>>>> Thanks, >>>>>> Zhu Zhu >>>>>> >>>>>> Steven Wu <stevenz...@gmail.com> 于2019年9月20日周五 上午7:11写道： >>>>>> >>>>>>> We do use config like "restart-strategy: >>>>>>> org.foobar.MyRestartStrategyFactoryFactory". Mainly to add additional >>>>>>> metrics than the Flink provided ones. >>>>>>> >>>>>>> On Thu, Sep 19, 2019 at 4:50 AM Zhu Zhu <reed...@gmail.com> wrote: >>>>>>> >>>>>>>> Thanks everyone for the input. >>>>>>>> >>>>>>>> The RestartStrategy customization is not recognized as a public >>>>>>>> interface as it is not explicitly documented. >>>>>>>> As it is not used from the feedbacks of this survey, I'll conclude >>>>>>>> that we do not need to support customized RestartStrategy for the new >>>>>>>> scheduler in Flink 1.10 >>>>>>>> >>>>>>>> Other usages are still supported, including all the strategies and >>>>>>>> configuring ways described in >>>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/task_failure_recovery.html#restart-strategies >>>>>>>> . >>>>>>>> >>>>>>>> Feel free to share in this thread if you has any concern for it. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Zhu Zhu >>>>>>>> >>>>>>>> Zhu Zhu <reed...@gmail.com> 于2019年9月12日周四 下午10:33写道： >>>>>>>> >>>>>>>>> Thanks Oytun for the reply! >>>>>>>>> >>>>>>>>> Sorry for not have stated it clearly. When saying "customized >>>>>>>>> RestartStrategy", we mean that users implement an >>>>>>>>> *org.apache.flink.runtime.executiongraph.restart.RestartStrategy* >>>>>>>>> by themselves and use it by configuring like "restart-strategy: >>>>>>>>> org.foobar.MyRestartStrategyFactoryFactory". >>>>>>>>> >>>>>>>>> The usage of restart strategies you mentioned will keep working >>>>>>>>> with the new scheduler. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Zhu Zhu >>>>>>>>> >>>>>>>>> Oytun Tez <oy...@motaword.com> 于2019年9月12日周四 下午10:05写道： >>>>>>>>> >>>>>>>>>> Hi Zhu, >>>>>>>>>> >>>>>>>>>> We are using custom restart strategy like this: >>>>>>>>>> >>>>>>>>>> environment.setRestartStrategy(failureRateRestart(2, >>>>>>>>>> Time.minutes(1), Time.minutes(10))); >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --- >>>>>>>>>> Oytun Tez >>>>>>>>>> >>>>>>>>>> *M O T A W O R D* >>>>>>>>>> The World's Fastest Human Translation Platform. >>>>>>>>>> oy...@motaword.com — www.motaword.com >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Sep 12, 2019 at 7:11 AM Zhu Zhu <reed...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi everyone, >>>>>>>>>>> >>>>>>>>>>> I wanted to reach out to you and ask how many of you are using a >>>>>>>>>>> customized RestartStrategy[1] in production jobs. >>>>>>>>>>> >>>>>>>>>>> We are currently developing the new Flink scheduler[2] which >>>>>>>>>>> interacts with restart strategies in a different way. We have to >>>>>>>>>>> re-design >>>>>>>>>>> the interfaces for the new restart strategies (so called >>>>>>>>>>> RestartBackoffTimeStrategy). Existing customized RestartStrategy >>>>>>>>>>> will not >>>>>>>>>>> work any more with the new scheduler. >>>>>>>>>>> >>>>>>>>>>> We want to know whether we should keep the way >>>>>>>>>>> to customized RestartBackoffTimeStrategy so that existing customized >>>>>>>>>>> RestartStrategy can be migrated. >>>>>>>>>>> >>>>>>>>>>> I'd appreciate if you can share the status if you are >>>>>>>>>>> using customized RestartStrategy. That will be valuable for use to >>>>>>>>>>> make >>>>>>>>>>> decisions. >>>>>>>>>>> >>>>>>>>>>> [1] >>>>>>>>>>> https://ci.apache.org/projects/flink/flink-docs-master/dev/task_failure_recovery.html#restart-strategies >>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-10429 >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Zhu Zhu >>>>>>>>>>> >>>>>>>>>>
