otterc commented on a change in pull request #35085:
URL: https://github.com/apache/spark/pull/35085#discussion_r789242528
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -2742,6 +2743,16 @@ private[spark] object Utils extends Logging {
new File(path.getAbsolutePath + "." + UUID.randomUUID())
}
+ /**
+ * Creates a file with group write permission.
+ */
+ def createFileAsGroupWritable(file: File): Unit = {
+ val perms = PosixFilePermissions.fromString("rw-rw----")
+ val path = file.toPath
+ Files.createFile(path)
+ Files.setPosixFilePermissions(path, perms)
Review comment:
In our env the shuffle service is run as `yarn:yarn`.
Below is the explanation for push-based shuffle dirs:
> This would create mergeDir as bob:hadoop with mode rwxr-s--- and
mergeDir/00 as bob:hadoop rwxrwx---.
In our env, the merge directory and its subdirs are owned by `<user>:yarn`
with permission `rwxrwx---`. Example:
`drwxrwx--- 2 chsingh yarn 20480 Jan 20 22:38 merge_manager`
The merged files under the merge dirs are owned by `yarn:yarn` but have
permissions `rw-rw-r--`. Example:
```
-rw-rw-r-- 1 yarn yarn 40 Jan 20 22:39
mergedShuffle_application_1632281309592_16957571_0_1252.index
-rw-rw-r-- 1 yarn yarn 40 Jan 20 22:39
mergedShuffle_application_1632281309592_16957571_0_1245.index
-rw-rw-r-- 1 yarn yarn 632 Jan 20 22:39
mergedShuffle_application_1632281309592_16957571_0_1208.meta
-rw-rw-r-- 1 yarn yarn 40 Jan 20 22:39
mergedShuffle_application_1632281309592_16957571_0_1208.index
-rw-rw-r-- 1 yarn yarn 40 Jan 20 22:39
mergedShuffle_application_1632281309592_16957571_0_1204.index
-rw-rw-r-- 1 yarn yarn 40 Jan 20 22:39
mergedShuffle_application_1632281309592_16957571_0_1200.index
-rw-rw-r-- 1 yarn yarn 632 Jan 20 22:39
mergedShuffle_application_1632281309592_16957571_0_1262.meta
```
Since these files are created by shuffle service, they are created by the
default permission. However these files are still not readable by any user
other than `chsingh` because the parent directory is not accessible by anyone
else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
