[ https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017758#comment-17017758 ]
Dongjoon Hyun edited comment on SPARK-30325 at 1/17/20 7:15 AM: ---------------------------------------------------------------- Hi, All. Since there is no UT, it's difficult to validate. According to the commit history, this bug seems to exist with SPARK-24755 and before it. Could you update `Affects Version/s:` more if it does? For example, 2.3.4 and 2.2.3? was (Author: dongjoon): Hi, All. Since there is no UT, it's difficult to validate. According to the commit history, this bug seems to exist with SPARK-24755 and before it. Could you update `Affects Version/s:` more? For example, 2.3.4 and 2.2.3? > markPartitionCompleted cause task status inconsistent > ----------------------------------------------------- > > Key: SPARK-30325 > URL: https://issues.apache.org/jira/browse/SPARK-30325 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4 > Reporter: haiyangyu > Assignee: haiyangyu > Priority: Major > Fix For: 2.4.5, 3.0.0 > > Attachments: image-2019-12-21-17-11-38-565.png, > image-2019-12-21-17-15-51-512.png, image-2019-12-21-17-16-40-998.png, > image-2019-12-21-17-17-42-244.png > > > h3. Corner case > The bugs occurs in the coren case as follows: > # The stage occurs for fetchFailed and some task hasn't finished, scheduler > will resubmit a new stage as retry with those unfinished tasks. > # The unfinished task in origin stage finished and the same task on the new > retry stage hasn't finished, it will mark the task partition on the new retry > stage as succesuful. !image-2019-12-21-17-11-38-565.png|width=427,height=154! > # The executor running those 'successful task' crashed, it cause > taskSetManager run executorLost to rescheduler the task on the executor, here > will cause copiesRunning decreate 1 twice, beause those 'successful task' are > not finished, the variable copiesRunning will decreate to -1 as result. > !image-2019-12-21-17-15-51-512.png|width=437,height=340!!image-2019-12-21-17-16-40-998.png|width=398,height=139! > # 'dequeueTaskFromList' will use copiesRunning equal 0 as reschedule basis > when rescheduler tasks, and now it is -1, can't to reschedule, and the app > will hung forever. !image-2019-12-21-17-17-42-244.png|width=366,height=282! -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org