Kimahriman commented on pull request #35085: URL: https://github.com/apache/spark/pull/35085#issuecomment-1011554123
I tested this out in my actual environment and realized that it didn't actually work because the context cleaner remove the map output before removing the shuffle from the block manager master. I switched the order of those and updated to test to user the cleaner to replicate and it sort of started working. I say sort of because I realized a major potential issue with this. We use a private secured Hadoop cluster and the shuffle files were not writable by the user running the shuffle service so they couldn't be deleted. I still need to do a little digging to see if the permissions for those files (via Spark or Hadoop) are configurable at all through the node managers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
