[
https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-30325.
---------------------------------
Fix Version/s: 3.0.0
Resolution: Fixed
Issue resolved by pull request 26975
[https://github.com/apache/spark/pull/26975]
> markPartitionCompleted cause task status inconsistent
> -----------------------------------------------------
>
> Key: SPARK-30325
> URL: https://issues.apache.org/jira/browse/SPARK-30325
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.0, 2.4.4
> Reporter: haiyangyu
> Priority: Major
> Fix For: 3.0.0
>
> Attachments: image-2019-12-21-17-11-38-565.png,
> image-2019-12-21-17-15-51-512.png, image-2019-12-21-17-16-40-998.png,
> image-2019-12-21-17-17-42-244.png
>
>
> h3. Corner case
> The bugs occurs in the coren case as follows:
> # The stage occurs for fetchFailed and some task hasn't finished, scheduler
> will resubmit a new stage as retry with those unfinished tasks.
> # The unfinished task in origin stage finished and the same task on the new
> retry stage hasn't finished, it will mark the task partition on the new retry
> stage as succesuful. !image-2019-12-21-17-11-38-565.png|width=427,height=154!
> # The executor running those 'successful task' crashed, it cause
> taskSetManager run executorLost to rescheduler the task on the executor, here
> will cause copiesRunning decreate 1 twice, beause those 'successful task' are
> not finished, the variable copiesRunning will decreate to -1 as result.
> !image-2019-12-21-17-15-51-512.png|width=437,height=340!!image-2019-12-21-17-16-40-998.png|width=398,height=139!
> # 'dequeueTaskFromList' will use copiesRunning equal 0 as reschedule basis
> when rescheduler tasks, and now it is -1, can't to reschedule, and the app
> will hung forever. !image-2019-12-21-17-17-42-244.png|width=366,height=282!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
