Everybody, Thank you for the quick response.
Yes, we inadvertently used the -d/--drain flag when stopping the job. We were not aware that it would cause a MAX_WATERMARK to roll through our system. MAX_WATERMARKS are catastrophic for the event time timers we have in our system. We know now never to use -d again for this situation. Again, thank you. -Marco On Thu, Sep 23, 2021 at 11:01 PM JING ZHANG <beyond1...@gmail.com> wrote: > Hi Macro, > Do you specified drain flag when stop a job with a savepoint? > If the --drain flag is specified, then a MAX_WATERMARK will be emitted > before the last checkpoint barrier. > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/cli/#stopping-a-job-gracefully-creating-a-final-savepoint > > Best, > JING ZHANG > > Marco Villalobos <mvillalo...@kineteque.com> 于2021年9月24日周五 下午12:54写道： > >> Something strange happened today. >> When we tried to shutdown a job with a savepoint, the watermarks became >> equal to 2^63 - 1. >> >> This caused timers to fire indefinitely and crash downstream systems with >> overloaded untrue data. >> >> We are using event time processing with Kafka as our source. >> >> It seems impossible for a watermark to be that large. >> >> I know its possible stream with a batch execution mode. But this was >> stream processing. >> >> What can cause this? Is this normal behavior when creating a savepoint? >> >
