Kimahriman commented on a change in pull request #35085:
URL: https://github.com/apache/spark/pull/35085#discussion_r795822134
##########
File path: core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala
##########
@@ -94,7 +95,13 @@ private[spark] class DiskBlockManager(
} else {
val newDir = new File(localDirs(dirId), "%02x".format(subDirId))
if (!newDir.exists()) {
- Files.createDirectory(newDir.toPath)
+ // SPARK-37618: Create dir as group writable so files within can be
deleted by the
+ // shuffle service
+ val path = newDir.toPath
+ Files.createDirectory(path)
+ val currentPerms = Files.getPosixFilePermissions(path)
+ currentPerms.add(PosixFilePermission.GROUP_WRITE)
Review comment:
Ok I figured it out. When you use `mkdir -m<perm>`, one of two things
happens:
- If `<perm>` meets the current `umask`, then the `mkdir` system call can
create the directory with the permission you tell it
- If `<perm>` doesn't meet the current `umask`, then a `mkdir` system call
will create the directory and follow it up with a `chmod` system call to set
the permissions to what you tell it
Looks like the default umask for a normal linux user is `0002`, so that's
why your example worked and it worked when I tried locally as well. If you set
the umask to `0022`, that's when you'll see the inner directories lose the
`setgid` bit. And the umask for secure yarn processes is 0027.
The chmod call is what doesn't really follow the man page behavior
correctly. It states:
```
Setuid and Setgid Bits
chmod clears the set-group-ID bit of a regular file if the file's group ID
does not match the user's effective group ID or one of the user's supplementary
group IDs, unless the user has appropriate privileges. Additional restrictions
may cause the set-user-ID and set-group-ID bits of MODE or RFILE to be ignored.
This behavior depends on the policy and functionality of the underlying chmod
system call. When in doubt, check the underlying system behavior.
chmod preserves a directory's set-user-ID and set-group-ID bits unless you
explicitly specify otherwise. You can set or clear the bits with symbolic modes
like u+s and g-s, and you can set (but not clear) the bits with a numeric mode.
```
It seems to be the first part of the description for "regular files" is
what's happening here. If the user you are running `chmod` isn't in the group
of the folder you're trying to `chmod`, then modifying the group write bit ends
up clearing the setgid bit. If you `strace` the system calls, you see it try to
`fchmod 02770`, but this ends up removing the setgid bit in this case for some
reason (based on the man page it sounds like a Linux bug to me)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
