Re: post-checkpoint watermark out of sync with event stream?

2020-04-15 Thread Aljoscha Krettek
Hi Cliff, On 14.04.20 19:29, Cliff Resnick wrote I'm wondering how this could be possible. The only explanation I can think of is: 4. on "endTime" timer key state is purged. 5 --- job fail --- 6. job restarted on 2.5 hour old Savepoint 7. watermark regresses (?) from "endTime" watermark. 8.

post-checkpoint watermark out of sync with event stream?

2020-04-14 Thread Cliff Resnick
We have an event-time pipeline that uses a ProcessFunction to accept events with an allowed lateness of a number of days. We a BoundedOutOfOrdernessTimestampExtractor and our event stream has a long tail that occasionally exceeds our allowed lateness, in which case we drop the events. The logic