[
https://issues.apache.org/jira/browse/SPARK-50288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-50288.
----------------------------------
Resolution: Not A Problem
Resolving as Not A Problem — the described behavior is working as designed. The
ticket is typed as a Question and asks why executors of a failed/retried stage
stay alive and what config forces them to terminate before retry. This is
provably works-as-designed, confirmed by code at HEAD (e9d2378b5a27):
TaskSetManager.scala:170-175 states TaskSetManagers enter the zombie state on
retry and remain there to "continue to track and account for the running
tasks," with a TODO to eventually kill them; DAGScheduler.scala:2682 in the
FetchFailed handler has "// TODO: Cancel running tasks in the failed stage --
cf. SPARK-17064." Because the old attempts (e.g. partitions 371/200) keep
running, their executors naturally stay alive — exactly the reporter's
observation, and by des
Please reopen with a reproducer against a supported version if you believe this
is a bug.
> Executors of stage failed still alive even through stage has been retried
> -------------------------------------------------------------------------
>
> Key: SPARK-50288
> URL: https://issues.apache.org/jira/browse/SPARK-50288
> Project: Spark
> Issue Type: Question
> Components: Shuffle, Spark Core
> Affects Versions: 3.3.0, 3.5.2
> Reporter: Yu-Ting LIN
> Priority: Major
> Attachments: stage19_stage19retry1.png, trigger_duplicate_process.png
>
>
> We are executing spark dataframe API with foreachPartition and we observed
> something that we are not able to explain.
> In the figure that we provided, you can see that stage 19 has been retried
> due to some ShuffleOutputNotFound error and stage 19 has been retried 1.
> !stage19_stage19retry1.png|width=1150,height=603!
> However, we found that there are still some executors allocated for stage 19
> in state RUNNING e.g. partition 371 and 200. In addition, as not yet
> finished, partition 371 and 200 have been resubmitted again in stage 19
> (retry 1).
> !trigger_duplicate_process.png|width=1144,height=402!
> Is there any configuration can help us make sure that all executor from the
> stage failed will be terminated before retry staged been triggered.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
