Kimahriman commented on a change in pull request #35085:
URL: https://github.com/apache/spark/pull/35085#discussion_r787862613
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2742,6 +2743,16 @@ private[spark] object Utils extends Logging {
new File(path.getAbsolutePath + "." + UUID.randomUUID())
}
+ /**
+ * Creates a file with group write permission.
+ */
+ def createFileAsGroupWritable(file: File): Unit = {
+ val perms = PosixFilePermissions.fromString("rw-rw----")
+ val path = file.toPath
+ Files.createFile(path)
+ Files.setPosixFilePermissions(path, perms)
Review comment:
Well sort of wrong again. At least in my environment, the setgid bit
still gets removed when creating the dir with `mkdir -m`. I don't know if this
is true on all linux systems, but it appears of the user trying to `chmod` a
folder or file (even though the man page just says file), has to be in the
group of the file in order to set or keep the setgid bit during a chmod call.
Otherwise if you set the group write bit, it will remove the setgid bit. This
caused all the shuffle blocks to not have the group of the yarn service (and
therefore shuffle service), so the shuffle service couldn't read the shuffle
files anymore.
After playing around, I was able to find a workaround by doing `umask 002 &&
mkdir <path>`, but that's getting more and more specific to linux systems. I
can't think of any other way though, so definitely would love to hear some
thoughts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
