alisha-1000 opened a new pull request, #41448: URL: https://github.com/apache/beam/pull/41448
Addresses #24662. #24662 reports that with `@RequiresStableInput` in the Flink runner, the output watermark can pass the output timestamp of a timer that has fired but is still buffered. I couldn't reproduce this on current master: - `BufferingDoFnRunner.onTimer()` adds the timer's output timestamp to the buffer's watermark hold. - When a timer fires, `onEventTime` starts a bundle, and `maybeEmitWatermark` doesn't emit a watermark while that bundle is open. - When the watermark is emitted later, `getEffectiveInputWatermark()` already applies `bufferingDoFnRunner.getOutputWatermarkHold()`, so the buffered timer is held correctly. This PR adds a regression test so the behavior stays covered: `DoFnOperatorTest.testExactlyOnceBufferingOutputWatermarkHoldsForBufferedTimers` 1. A keyed `@RequiresStableInput` DoFn processes an element at 10 and sets an event-time timer at 50. The checkpoint completes, so the element is processed. 2. Watermark 100 fires the timer into the buffer. After the bundle finishes, the output watermark is held at 50. 3. After the next checkpoint completes, the timer output is emitted at 50 and the watermark advances to 100. 4. The output is exactly `[Watermark 50, KV(key, timer) @50, Watermark 100]`, and no watermark above 50 is emitted before the timer output. The test passes on master. It guards against a regression and doesn't fix a bug. I ran the operator tests on `:runners:flink:1.19` (base sources) and `:runners:flink:2.2`. I didn't add a portable `ExecutableStageDoFnOperator` test because buffering a timer through the mocked SDK harness needs a lot of setup. I'm happy to add one if you think it's worth it. @je-ik, since you filed the issue and know this code, could you take a look? If you agree it's already fixed on master, #24662 could be closed once this is merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
