Kimahriman commented on PR #42301: URL: https://github.com/apache/spark/pull/42301#issuecomment-1751368798
> Yeah, this seems to be working around a specific resource manager in a corner case, I'm not sure if this is a good solution. I agree I don't like adding a config, I think it should be just globally changed to use the java tmp dir instead of the configured local dirs, but I don't know what the behavior is of k8s or standalone for how local directories work. I think it is currently incorrect behavior on YARN because it's using application level directories for executor/container specific temp files that should be deleted when that executor/container is finished or fails. It's not like shuffle data with a shuffle service that _should_ outlive the executor/container. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
