alisha-1000 opened a new pull request, #41514:
URL: https://github.com/apache/beam/pull/41514

   Fixes #41512
   
   `KeyedBufferingElementsHandler.getElements()` called `setCurrentKey`
   inside a lazy `flatMap`, but `BufferingDoFnRunner` consumed the stream
   through `.iterator()`, which buffers a whole key group first. All
   `setCurrentKey` calls for a key group ran before any element was
   processed, so every buffered element and timer was replayed under the
   last key of the group.
   
   Changes:
   - `BufferingElementsHandler.getElements()` now returns an `Iterator`.
     The keyed implementation moves to the next key only after the
     previous key's elements are consumed, and sets the current key right
     before returning each element.
   - The non-keyed implementation returns the state list's iterator.
   - The checkpoint state format is unchanged (same coder, state names
     and descriptors).
   - New test in `DoFnOperatorTest`: two keys in one key group, each
     setting a timer. Before this change one timer output was lost.
   
   Tested `DoFnOperatorTest` and `BufferingDoFnRunnerTest` on
   `:runners:flink:1.19` and `:runners:flink:2.2`.
   
   Not changed here: `KeyedPushedBackElementsHandler#getElements()` uses
   the same `flatMap` + `setCurrentKey` pattern and is consumed via
   `.iterator()` in `DoFnOperator`. It may have the same problem but is
   not changed or tested in this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to