tkaymak commented on issue #40517: URL: https://github.com/apache/beam/issues/40517#issuecomment-6130511840
Thanks for taking this. Two things to line up: 1. Clearing `beamWatermark` turns the next visit of that key into a first visit, which filters late data against `TIMESTAMP_MIN_VALUE` (contract comment in `timerInternals`, #40446). A late element for a cleared key would then be accepted instead of dropped. Please cover that case in the test and say in the PR how you handle it, for example by keeping a much smaller marker or by accepting it as part of #40446. 2. `BeamStatefulProcessor` is about to change shape: #40516 (open) switches the Spark grouping key to encoded bytes, and the next PR for windowed GroupByKey turns it into an abstract base with a ParDo and a GroupByKey subclass. The cleanup belongs in the base, so building on top of #40516 and keeping the change inside `handleInputRows` and `handleExpiredTimer` keeps the conflict small. I will link the GroupByKey PR here when it is open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
