Chao Sun created SPARK-58879:
--------------------------------
Summary: Graceful dynamic allocation can decommission executors
with newly assigned tasks
Key: SPARK-58879
URL: https://issues.apache.org/jira/browse/SPARK-58879
Project: Spark
Issue Type: Bug
Components: Scheduler, Spark Core
Affects Versions: 4.0.0, 3.5.0
Reporter: Chao Sun
Assignee: Chao Sun
h3. Problem
SPARK-9552 protected ordinary dynamic-allocation scale-down from removing an
executor that became busy after it was identified as idle. The
graceful-decommission path introduced by SPARK-31198 does not perform the
equivalent check. When both {{spark.dynamicAllocation.enabled}} and
{{spark.decommission.enabled}} are enabled, an idle-timeout candidate can
therefore be accepted for decommissioning even after the scheduler has assigned
it new work.
h3. Example interleaving
# The allocation manager identifies executor E as an idle-timeout candidate.
# A scheduler offer assigns a task to E. {{TaskSchedulerImpl}} records the
assignment before the backend sends {{LaunchTask}}.
# The allocation manager requests graceful decommissioning of E.
{{CoarseGrainedSchedulerBackend.decommissionExecutors}} checks that E is
active, but does not recheck {{scheduler.isExecutorBusy(E)}}.
# E is marked for decommissioning even though it now has assigned work. The
task can reach or continue on E after decommissioning starts.
This can cause unnecessary task retries. With storage decommissioning enabled,
a task that tries to save a non-broadcast block after BlockManager
decommissioning begins can fail. This is a core scheduler race; it does not
depend on Spark Connect or a particular Kubernetes pod allocator.
h3. Expected behavior and proposed fix
Ordinary idle scale-down should recheck the scheduler's current task
assignments atomically with making an executor unavailable for further offers.
Add an idle-only graceful-decommission entry point and use it from the core
allocation manager. In the coarse-grained backend, hold the existing
scheduler/backend locks while filtering active, idle executors and invoking the
existing decommission path. If assignment wins, scale-down must reject that
executor; if decommissioning wins, later offers must exclude it. Preserve the
existing unconditional path for explicit and infrastructure-initiated drains.
h3. Public source and related work
The issue is present in released Spark 3.5.0 and 4.0.0 and in current master at
{{75ac74bafec1f489224765041a19bec6bad36a9d}}.
* [Graceful admission checks active state
only|https://github.com/apache/spark/blob/75ac74bafec1f489224765041a19bec6bad36a9d/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L559-L575]
* [Non-forced kill already checks whether the executor is
busy|https://github.com/apache/spark/blob/75ac74bafec1f489224765041a19bec6bad36a9d/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L969-L974]
* [Core allocation manager's graceful
path|https://github.com/apache/spark/blob/75ac74bafec1f489224765041a19bec6bad36a9d/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L583-L590]
* [BlockManager write
guard|https://github.com/apache/spark/blob/75ac74bafec1f489224765041a19bec6bad36a9d/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L395-L400]
Related: SPARK-9552 and SPARK-31198.
A deterministic regression test should pause after a real scheduler has
assigned a task but before {{LaunchTask}} is delivered, then attempt idle
decommissioning. It should also exercise the reverse ordering and confirm that
explicit decommissioning of a busy executor remains supported.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
