alisha-1000 opened a new pull request, #41514:
URL: https://github.com/apache/beam/pull/41514
Fixes #41512
`KeyedBufferingElementsHandler.getElements()` called `setCurrentKey`
inside a lazy `flatMap`, but `BufferingDoFnRunner` consumed the stream
through `.iterator()`, which buffers a whole key group first. All
`setCurrentKey` calls for a key group ran before any element was
processed, so every buffered element and timer was replayed under the
last key of the group.
Changes:
- `BufferingElementsHandler.getElements()` now returns an `Iterator`.
The keyed implementation moves to the next key only after the
previous key's elements are consumed, and sets the current key right
before returning each element.
- The non-keyed implementation returns the state list's iterator.
- The checkpoint state format is unchanged (same coder, state names
and descriptors).
- New test in `DoFnOperatorTest`: two keys in one key group, each
setting a timer. Before this change one timer output was lost.
Tested `DoFnOperatorTest` and `BufferingDoFnRunnerTest` on
`:runners:flink:1.19` and `:runners:flink:2.2`.
Not changed here: `KeyedPushedBackElementsHandler#getElements()` uses
the same `flatMap` + `setCurrentKey` pattern and is consumed via
`.iterator()` in `DoFnOperator`. It may have the same problem but is
not changed or tested in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]