David Peacock created FLINK-41115:
-------------------------------------
Summary: Checkpoint cleanup can starve AdaptiveScheduler
execution-graph creation
Key: FLINK-41115
URL: https://issues.apache.org/jira/browse/FLINK-41115
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 2.3.0
Environment: AdaptiveScheduler; Kubernetes; remote checkpoint storage
(GCS in the observed cases); two general JobManager IO workers. Local
regression exercises one and two workers.
Reporter: David Peacock
AdaptiveScheduler dispatches background execution-graph construction and
initial state assignment onto the JobManager's general IO executor. Checkpoint
cleanup also uses this executor. When cleanup occupies all its workers,
adaptive rescaling/recovery can remain in `CreatingExecutionGraph` even though
graph construction itself is inexpensive and the cleanup is not a prerequisite
for restoring retained state.
This is distinct from FLINK-26590, which concerns delayed initialization of
**new checkpoints**. This issue concerns rebuilding the **execution graph
during recovery/rescaling**, while preserving the existing relationship between
checkpoint-trigger initialization and cleanup.
h4. Observed evidence
On a 2.3.0-based downstream build, two rescale attempts spent approximately
**127.4s and 104.4s between dispatch and graph-worker entry**, followed by
approximately **0.32s executing the graph supplier**. Eight thread-dump samples
during those intervals showed both general IO workers in GCS deletion paths;
several exposed `SharedStateRegistryImpl.AsyncDisposalRunnable` callers.
h4. Reproduction
A downstream MiniCluster regression reproduces the mechanism deterministically:
# Run a stateful streaming job with AdaptiveScheduler, one or two general IO
workers, and two retained checkpoints.
# Complete checkpoints containing generation-specific file-backed state. During
subsumption, block enough eligible shared/private deletion calls to occupy
every general IO worker.
# Request a parallelism change while keeping the deletion calls blocked.
# Observe that replacement tasks cannot restore state and make progress until
deletion is released.
# Isolate only graph-supplier dispatch and repeat: physical tasks restore the
latest state and advance counters before deletion is released. A new checkpoint
still waits on general IO; after release, cleanup drains and checkpointing
resumes.
Routing the supplier back to general IO makes both worker-count cases fail on
restored-task progress. The regression can accompany the proposed upstream
change.
h4. Expected behavior
Adaptive graph reconstruction should not be starved solely because unrelated
checkpoint cleanup occupies the general IO executor. Existing
checkpoint-trigger/cleanup feedback, retained-state ownership and lifecycle
ordering should remain intact.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
