Hi Guowei, I could provide a case that I have encountered which timers to fire indefinitely when doing drain savepoint. After an event timer is triggered, it registers another event timer whose value equals the value of triggered timer plus an interval time. If a MAX_WATERMARK comes, the timer is triggered, then registers another timer and forever. I'm not sure whether Macro meets a similar problem.
Best, JING ZHANG Guowei Ma <guowei....@gmail.com> 于2021年9月24日周五 下午4:01写道： > Hi Macro > > Indeed, as mentioned by JING, if you want to drain when triggering > savepoint, you will encounter this MAX_WATERMARK. > But I have a problem. In theory, even with MAX_WATERMARK, there will not > be an infinite number of timers. And these timers should be generated by > the application code. > You can share your code if it is convenient for you. > > Best, > Guowei > > > On Fri, Sep 24, 2021 at 2:02 PM JING ZHANG <beyond1...@gmail.com> wrote: > >> Hi Macro, >> Do you specified drain flag when stop a job with a savepoint? >> If the --drain flag is specified, then a MAX_WATERMARK will be emitted >> before the last checkpoint barrier. >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/cli/#stopping-a-job-gracefully-creating-a-final-savepoint >> >> Best, >> JING ZHANG >> >> Marco Villalobos <mvillalo...@kineteque.com> 于2021年9月24日周五 下午12:54写道： >> >>> Something strange happened today. >>> When we tried to shutdown a job with a savepoint, the watermarks became >>> equal to 2^63 - 1. >>> >>> This caused timers to fire indefinitely and crash downstream systems >>> with overloaded untrue data. >>> >>> We are using event time processing with Kafka as our source. >>> >>> It seems impossible for a watermark to be that large. >>> >>> I know its possible stream with a batch execution mode. But this was >>> stream processing. >>> >>> What can cause this? Is this normal behavior when creating a savepoint? >>> >>
