Gen Luo created FLINK-39680:
-------------------------------
Summary: Enable unaligned checkpoint for pointwise connection by
FORCE_UNALIGNED
Key: FLINK-39680
URL: https://issues.apache.org/jira/browse/FLINK-39680
Project: Flink
Issue Type: Improvement
Reporter: Gen Luo
Unaligned checkpoints on pointwise edges are currently prohibited by
FLINK-21936 and FLINK-21945 to ensure data consistency during parallelism
changes. This restriction is necessary when a keyed stream or keyed source
feeds into a pointwise edge, as channel state is redistributed upon recovery
with changed parallelism. This breaks the guarantee that records of the same
key reach the same downstream task, potentially compromising logic that relies
on key ordering.
However, there are valid scenarios where key ordering is irrelevant, but
unaligned checkpoints are essential. For instance, in bounded streaming jobs
processing datasets like images, each record may take seconds or minutes to
process. This can block checkpoint barriers, making aligned checkpoints
infeasible within expected timeouts.
Since Flink cannot automatically determine whether a job requires strict key
ordering, we should preserve the current default behavior. Instead, I propose
allowing users to explicitly opt-in to unaligned checkpoints for such cases.
The existing configuration
`execution.checkpointing.unaligned.forced`—originally introduced to support
iterative jobs—can be leveraged for this purpose.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
