[
https://issues.apache.org/jira/browse/SPARK-58446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58446:
-----------------------------------
Labels: pull-request-available (was: )
> Dynamic allocation can remain at zero executors after TaskStart events arrive
> after StageCompleted
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-58446
> URL: https://issues.apache.org/jira/browse/SPARK-58446
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.3.0
> Reporter: Jiwon Park
> Priority: Major
> Labels: pull-request-available
>
> When dynamic allocation is enabled with
> spark.dynamicAllocation.minExecutors=0, an application can stop requesting
> executors after TaskStart events are processed after StageCompleted for the
> same stage attempt.
> The completed stage can contribute a negative pending task count. After all
> executors are removed by idle timeout, later jobs can remain pending because
> the target executor count stays at 0 and no request is sent to the cluster
> manager.
> Reproduction event sequence:
> StageSubmitted -> TaskStart -> StageCompleted -> late TaskStart -> TaskEnd ->
> new StageSubmitted
> Expected behavior:
> Completed stage attempts do not contribute to pending task counts, and
> submitting a new stage increases the executor target.
> Actual behavior:
> The pending task count can remain negative, causing the executor target to
> stay at 0 and preventing new tasks from starting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
