[
https://issues.apache.org/jira/browse/FLINK-41115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41115:
-----------------------------------
Labels: pull-request-available (was: )
> Checkpoint cleanup can starve AdaptiveScheduler execution-graph creation
> ------------------------------------------------------------------------
>
> Key: FLINK-41115
> URL: https://issues.apache.org/jira/browse/FLINK-41115
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 2.3.0
> Environment: AdaptiveScheduler; Kubernetes; remote checkpoint storage
> (GCS in the observed cases); two general JobManager IO workers. Local
> regression exercises one and two workers.
> Reporter: David Peacock
> Priority: Major
> Labels: pull-request-available
>
> AdaptiveScheduler dispatches background execution-graph construction and
> initial state assignment onto the JobManager's general IO executor.
> Checkpoint cleanup also uses this executor. When cleanup occupies all its
> workers, adaptive rescaling/recovery can remain in `CreatingExecutionGraph`
> even though graph construction itself is inexpensive and the cleanup is not a
> prerequisite for restoring retained state.
> This is distinct from FLINK-26590, which concerns delayed initialization of
> **new checkpoints**. This issue concerns rebuilding the **execution graph
> during recovery/rescaling**, while preserving the existing relationship
> between checkpoint-trigger initialization and cleanup.
> h4. Observed evidence
> On a 2.3.0-based downstream build, two rescale attempts spent approximately
> **127.4s and 104.4s between dispatch and graph-worker entry**, followed by
> approximately **0.32s executing the graph supplier**. Eight thread-dump
> samples during those intervals showed both general IO workers in GCS deletion
> paths; several exposed `SharedStateRegistryImpl.AsyncDisposalRunnable`
> callers.
> h4. Reproduction
> A downstream MiniCluster regression reproduces the mechanism
> deterministically:
> # Run a stateful streaming job with AdaptiveScheduler, one or two general IO
> workers, and two retained checkpoints.
> # Complete checkpoints containing generation-specific file-backed state.
> During subsumption, block enough eligible shared/private deletion calls to
> occupy every general IO worker.
> # Request a parallelism change while keeping the deletion calls blocked.
> # Observe that replacement tasks cannot restore state and make progress until
> deletion is released.
> # Isolate only graph-supplier dispatch and repeat: physical tasks restore the
> latest state and advance counters before deletion is released. A new
> checkpoint still waits on general IO; after release, cleanup drains and
> checkpointing resumes.
> Routing the supplier back to general IO makes both worker-count cases fail on
> restored-task progress. The regression can accompany the proposed upstream
> change.
> h4. Expected behavior
> Adaptive graph reconstruction should not be starved solely because unrelated
> checkpoint cleanup occupies the general IO executor. Existing
> checkpoint-trigger/cleanup feedback, retained-state ownership and lifecycle
> ordering should remain intact.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
