[GitHub] [spark] otterc commented on pull request #40393: [SPARK-40082] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks
otterc commented on PR #40393: URL: https://github.com/apache/spark/pull/40393#issuecomment-1474749275 @Stove-hust The main change in `DAGScheduler` looks good to me. Basically, [here](https://github.com/apache/spark/blob/11c9838283e98d5ebe6ce13b85e26217494feef2/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L762) we also check whether the parent stage is finalized and if it is not we submit that. The reason the parent stage is not getting finalized here is because it has no tasks. Will review the UT and take another look at the code next week. Thanks for addressing this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[GitHub] [spark] otterc commented on pull request #40393: [SPARK-40082] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks
otterc commented on PR #40393: URL: https://github.com/apache/spark/pull/40393#issuecomment-1474477575 One of the problems that I see is that a successful completion of speculative task will not trigger shuffle merge finalization of a stage that was marked failed but doesn't have any pending partitions. I think we need to address this as well with the fix. cc. @Stove-hust @mridulm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[GitHub] [spark] otterc commented on pull request #40393: [SPARK-40082] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks
otterc commented on PR #40393: URL: https://github.com/apache/spark/pull/40393#issuecomment-1470501033 @akpatnam25 @shuwang21 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[GitHub] [spark] otterc commented on pull request #40393: [SPARK-40082] Schedule mergeFinalize when push merge shuffleMapStage retry but no running tasks
otterc commented on PR #40393: URL: https://github.com/apache/spark/pull/40393#issuecomment-1469293356 @Stove-hust Haven't had a chance to look at it yet. I'll take a look at it this week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
