Ngone51 commented on issue #23871: [SPARK-23433][SPARK-25250] [CORE] Later created TaskSet should learn about the finished partitions URL: https://github.com/apache/spark/pull/23871#issuecomment-467449606 > can we address [#21131 (comment)](https://github.com/apache/spark/pull/21131#discussion_r203257926) as well? I revisit the discussion and code, I think maybe we could not remove `maybeFinishTaskSet()` there. Think about the case: Stage 0 has 5 partitions and there's an active TSM 0.1 and a zombie TSM 0.0 in TaskScheduler currently. TSM 0.1 has finished partition [0, 1, 2, 3] and no running tasks(may be blocked by resource offer or delay scheduling) at the moment a succeed task from TSM 0.0 finished. Thus, we have `successful[4]=true` for TSM 0.1 by calling `markPartitionCompleted`. So, if we do not call `maybeFinishTaskSet` in the `markPartitionCompleted`, TSM 0.1 can not finish at the end because it has no tasks to launch, which results in we could not call `maybeFinishTaskSet` at somewhere else. WDYT ? @cloud-fan
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
