Maximilian Michels created BEAM-10558:
-----------------------------------------
Summary: Flushing of buffered elements during checkpoint can stall
Key: BEAM-10558
URL: https://issues.apache.org/jira/browse/BEAM-10558
Project: Beam
Issue Type: Bug
Components: runner-flink
Reporter: Maximilian Michels
Assignee: Maximilian Michels
Elements are buffered during {{DoFnOperator#snapshotState}}, called as part of
a Flink checkpoint. This is necessary because flushing out elements in this
method call would alter the checkpoint barrier alignment. Optionally, elements
can be flushed out before the method call via the
{{finishBundleBeforeCheckpointing}} option which is turned off by default
because it can affect the checkpoint duration.
The buffer is flushed as part of starting a new bundle. A problem arises if no
new bundle will be started. For example, this can be the case if only a single
element (e.g. Impulse) is produced as part of a bundle during checkpointing.
Afterwards, when no bundle will be started due to another element arriving, or
a timer firing, the element will not be flushed from the buffer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
