tkaymak commented on PR #39576: URL: https://github.com/apache/beam/pull/39576#issuecomment-5362273244
PAssert update. Streaming PAssert now works in non global windows too, commit e991c5d7712. The earlier commit made PAssert resolve in the global window by emitting an end of stream sentinel that advances the watermark to TIMESTAMP_MAX_VALUE. Non global windows still failed with an empty assertion iterable. The cause was not the fixed windows themselves, PAssert rewindows everything into the global window and chains two GroupByKeys, and the late data filter in front of the second one judged the global window expired because LateDataUtils truncates garbage collection times to the end of the global window while the sentinel watermark sits one day past it. The pane flushed by the first GroupByKey was dropped on arrival at the second. The fix clamps the watermark used for arrival side expiry decisions to the end of the global window, through a delegating TimerInternals view in the step context. Timer firing and the ReduceFnRunner triggers keep the real watermark. Late data semantics for finite windows are unchanged, the clamp only takes effect once the watermark is already past the end of the global window, which only the sentinel can cause. Tests: the fixed windows PAssert case is restored, and both window modes now have negative tests proving a wrong expectation genuinely fails the pipeline. Full local streaming suites, 61 tests, 0 failures. Known remaining gap, documented rather than hidden: chained groupings in finite windows that are flushed only by the final sentinel would still drop their in flight panes. PAssert never builds that shape, it always rewindows to global first. The clean general solution is per operator output watermark holds, which is on the roadmap for the productionization phase. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
