rmcyang commented on pull request #34461: URL: https://github.com/apache/spark/pull/34461#issuecomment-965824279
Ok this makes sense to me - current behavior indeed seems not a very efficient way to handle the scenario of stage retry when the parent stage is determinate and the parent stage has merge data, which should not be ignored by that condition. Given SPARK-37023 is designated to fix the enableBatchFetch assertion failure bug for current behavior, shall we file another spark jira and implement the improvement there? @Ngone51 @mridulm @Victsm @zhouyejoe @otterc @venkata91 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
