Maximilian Michels created BEAM-8549:
----------------------------------------
Summary: Buffered state during checkpointing may be keyed
incorrectly
Key: BEAM-8549
URL: https://issues.apache.org/jira/browse/BEAM-8549
Project: Beam
Issue Type: Bug
Components: runner-flink
Reporter: Maximilian Michels
Assignee: Maximilian Michels
Fix For: 2.17.0
The current buffer logic for items emitted during checkpointing is faulty in
the sense that the buffer is partitioned on the output keys of the operator.
The key may be changed or even be dropped. Thus, the original key partitioning
will not be maintained which will cause checkpointing to fail.
An alternative solution would be BEAM-6733, but we may want to keep the current
buffering logic in place. The proposed solution here is to remove the keyed
buffering. The output buffer may then always be redistributed round-robin upon
restoring from a checkpoint. Note that this is fine because no assumption can
be made about the distribution of output elements of an operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
