Ngone51 commented on issue #24375: [SPARK-27474][CORE] try best to not submit tasks when the partitions are already completed URL: https://github.com/apache/spark/pull/24375#issuecomment-483635446 > Yes it's kind of a variant of #21131. This PR covers the corner case that, an active TaskSetManager hasn't been created when a previous task succeed. Does this pr really cover the corner case ? Do I miss something ? > That said, this PR tries to avoid the worst case described above. Even if we go through the event loop now, I don't think it will take a very long time that the task from the active TSM have already re-tried 3 times. Yeah, I think this pr could avoid the worst case event if we go through the event loop. But, doesn't #21131 could do it either ? What's the advantage compares to #21131 ? > Maybe we can combine the solution? e.g. when a task from zombie TSM completes, notify the active TSM immediately. Later the DAGScheduler notify the active TSM again, in case it was not created when the task from zombie TSM completed. I'm neutral on combining the solution since we're doing optimization(saving resource) instead of bug. Though, I'm wodering whether it would do a good help.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
