Mayur Bhosale created SPARK-45498:
-------------------------------------
Summary: Followup: Ignore task completion from old stage after
retrying indeterminate stages
Key: SPARK-45498
URL: https://issues.apache.org/jira/browse/SPARK-45498
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.0.0, 3.5.1
Reporter: Mayur Bhosale
With SPARK-45182, we added a fix for not letting laggard tasks of the older
attempts of the indeterminate stage from marking the partition has completed in
map output tracker.
When a task completes, DAG scheduler also notifies all the tasksets of the
stage about that partition being completed. Tasksets would not schedule such
task if they are not already scheduled. This is not correct for indeterminate
stage, since we want to re-run all the tasks on re-attempt
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
