Kimahriman commented on a change in pull request #35085:
URL: https://github.com/apache/spark/pull/35085#discussion_r795084112
##########
File path: core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala
##########
@@ -94,7 +95,13 @@ private[spark] class DiskBlockManager(
} else {
val newDir = new File(localDirs(dirId), "%02x".format(subDirId))
if (!newDir.exists()) {
- Files.createDirectory(newDir.toPath)
+ // SPARK-37618: Create dir as group writable so files within can be
deleted by the
+ // shuffle service
+ val path = newDir.toPath
+ Files.createDirectory(path)
+ val currentPerms = Files.getPosixFilePermissions(path)
+ currentPerms.add(PosixFilePermission.GROUP_WRITE)
Review comment:
From what I understand, `createDirWithPermission770` was made because it
was thought it couldn't be done through the Java API, but this works in my
environment through the Java API correctly. See later in that same thread for
that discussion:
https://github.com/apache/spark/pull/35085#discussion_r789858724
Because it might be possible to remove that helper after all (still waiting
to hear if there's some other reason create then set permission won't work in
other environments), I just stuck with the nio API.
From what I've found, losing the `setgid` bit isn't actually Java related.
`mkdir -p -m770` also removes the `setgid` bit because of the way the `chmod`
Linux system call works (in the operating systems I tested at least). Despite
the man page saying that it preservers the setgid bit on directories, the same
behavior that the man page lists for files seems to apply to directories. If
the user isn't in the group of the directory, it will remove the setgid bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
