Kimahriman commented on a change in pull request #35085:
URL: https://github.com/apache/spark/pull/35085#discussion_r795053646
##########
File path:
core/src/main/java/org/apache/spark/shuffle/sort/io/LocalDiskShuffleMapOutputWriter.java
##########
@@ -87,7 +86,7 @@ public ShufflePartitionWriter getPartitionWriter(int
reducePartitionId) throws I
}
lastPartitionId = reducePartitionId;
if (outputTempFile == null) {
- outputTempFile = Utils.tempFileWith(outputFile);
+ outputTempFile = blockResolver.createTempFile(outputFile);
Review comment:
Some of the context is here:
https://github.com/apache/spark/pull/35085#discussion_r788725899
Basically in a secure yarn environment, the `setgid` bit is set on all
directories so that it keeps the same group as the node manager and shuffle
service, but when we add the group write permission to the directory, it
removes the setgid bit, so shuffle files get created without the node manager
group with mode `rw-r-----`, and the shuffle service doesn't have permission to
read it. This creates the shuffle files world readable to the shuffle service
can still _read_ the files with this update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
