AngersZhuuuu commented on PR #43906: URL: https://github.com/apache/spark/pull/43906#issuecomment-1820111014
> @AngersZhuuuu thanks for digging into this, I think this a good fix, and [SPARK-39601](https://issues.apache.org/jira/browse/SPARK-39601) still may be a valid supplement for the same kind of issues - executor launches after the Driver is shutdown, then errors occur. > > * [SPARK-39601](https://issues.apache.org/jira/browse/SPARK-39601) aims to suppress the executor error in YARN allocator after the Driver is shutdown, which fixes the issue on the consumer side > * this PR aims to cancel executor allocation after the Driver is shutdown, which fixes the issue on the producer side, it could reduce the pressure of RM. But I'm not sure if such canceling could kill all executors without error reports, including the pending executors and the launching executors? If yes, [SPARK-39601](https://issues.apache.org/jira/browse/SPARK-39601) seems redundant, if not, [SPARK-39601](https://issues.apache.org/jira/browse/SPARK-39601) is valid too. This pr aims not to allocate new executor, https://issues.apache.org/jira/browse/SPARK-39601 can avoid pending allocation request cause app failed by `Max number of executor failures` when minExecutor value is small. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
