hiboyang opened a new pull request #31715: URL: https://github.com/apache/spark/pull/31715
### What changes were proposed in this pull request? Add Spark config "spark.shuffle.markFileLostOnExecutorLost" to not delete shuffle file on executor lost event ### Why are the changes needed? There are multiple work going on with disaggregated/remote shuffle service (e.g. LinkedIn shuffle, Facebook shuffle service, Uber shuffle service). In those systems, shuffle data will be stored on different server other than executor. Spark should not mark shuffle data lost when the executor is lost. We could add a Spark configuration to control this behavior. By default, Spark still mark shuffle file lost. For disaggregated/remote shuffle service, people could set the configure to not mark shuffle file lost. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Run Spark unit test ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
