cloud-fan edited a comment on issue #24375: [SPARK-25250][CORE] try best to not submit tasks when the partitions are already completed URL: https://github.com/apache/spark/pull/24375#issuecomment-483498369 Yes it's kind of a variant of #21131. This PR covers the corner case that, an active `TaskSetManager` hasn't been created when a previous task succeed. There is a corner case that is un-fixable: when a task from a zombie TSM completes, the active TSM has already submitted the task for the same partition. #21131 doesn't cover it, and this PR doesn't cover it either. But this PR does make it more likely to happen, because we go through the event loop, so the active TSM takes longer to know that a partition has completed. Maybe we can combine the solution? e.g. when a task from zombie TSM completes, notify the active TSM immediately. Later the `DAGScheduler` notify the active TSM again, in case it was not created when the task from zombie TSM completed.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
