pan3793 commented on PR #43906: URL: https://github.com/apache/spark/pull/43906#issuecomment-1819058067
@AngersZhuuuu thanks for digging into this, I think this a good fix, and SPARK-39601 still may be a valid supplement for the same kind of issues - executor launches after the Driver is shutdown, then errors occur. - SPARK-39601 aims to suppress the executor error in YARN allocator after the Driver is shutdown, which fixes the issue on the consumer side - this PR aims to cancel executor allocation after the Driver is shutdown, which fixes the issue on the producer side, it could reduce the pressure of RM. But I'm not sure if such canceling could kill all executors without error reports, including the pending executors and the launching executors? If yes, SPARK-39601 seems redundant, if not, SPARK-39601 is valid too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
