mridulm commented on PR #35683: URL: https://github.com/apache/spark/pull/35683#issuecomment-1089810368
@abhishekd0907 While it is possible to configure spark applications running on yarn not to use external shuffle service, it is much more common to have it enabled. Given the comment [above](https://github.com/apache/spark/pull/35683#issuecomment-1081453295) > If decommissioning is enabled, shuffle blocks are already migrated to another node before the executor exits. However, this is best effort. So shuffle data might be left if shuffle blocks migration is left incomplete or the executor doesn't exit cleanly. we do not want to move shuffle blocks around if decomissioning is enabled when external shuffle service is in use. You could explicitly scope this PR to the case when ESS use is not enabled - namely, add the additional check for `SHUFFLE_SERVICE_ENABLED` being disabled. Thoughts ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
