pgandhi999 edited a comment on issue #22806: [SPARK-25250][CORE] : Late zombie task completions handled correctly even before new taskset launched URL: https://github.com/apache/spark/pull/22806#issuecomment-465756704 I would like to disagree with you on the above @Ngone51 , the PR is an implementation of your proposal, but with small changes. > In `TaskSchedulerImpl` we maintain a map, e.g. called `stageIdToFinishedPartitions`. And each time we call `sched.markPartitionCompletedInAllTaskSets(stageId, tasks(index).partitionId, info)`, we do an extra thing, updating the finished partitionId into `stageIdToFinishedPartitions`. Then, when creating a new `TaskSetManager` In `TaskSchedulerImpl`, we always excludes `Task`s that corresponding to the finished partitions firstly by looking into `stageIdToFinishedPartitions.` In your proposal above, you have mentioned that we can look the map `stageIdToFinishedPartitions` and exclude those partitions which are finished while creating a new TSM. This was exactly the reason I had to come up with the PR in the first place as when the new TSM is created, the tasks from the previous stage attempt are still running so partitions are not yet marked as complete. I have basically come up with a solution to ensure that once the partition completes, other tasks running on the same partition fail once and then do not get rescheduled as it is happening currently. The problem with the old change was that we were not using a lock while calling `markPartitionCompleted()` from TaskSchedulerImpl, the new solution ensures that all calls happen within the lock and no new locks need to be created.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
