otterc commented on a change in pull request #32007:
URL: https://github.com/apache/spark/pull/32007#discussion_r614521383
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -315,6 +315,38 @@ private[spark] object Utils extends Logging {
dir.getCanonicalFile
}
+ /**
+ * Create a directory that is writable by the group.
+ * Grant 770 permission so the shuffle server can create subdirs/files
within the merge folder.
+ */
+ def createDirWith770(dirToCreate: File): Unit = {
+ var attempts = 0
+ val maxAttempts = MAX_DIR_CREATION_ATTEMPTS
+ var created: File = null
+ while (created == null) {
+ attempts += 1
+ if (attempts > maxAttempts) {
+ throw new IOException(
+ s"Failed to create directory ${dirToCreate.getAbsolutePath} after " +
+ s"${maxAttempts} attempts!")
+ }
+ try {
+ val builder = new ProcessBuilder().command(
+ "mkdir", "-m770", dirToCreate.getAbsolutePath)
Review comment:
@mridulm I've tried this last year and it doesn't work. It still creates
the directory with permission `750`. Internally we have this TODO as well for
this method.
```
* TODO: Find out why can't we create a dir using java api with permission
770
* Files.createDirectories(mergeDir.toPath,
PosixFilePermissions.asFileAttribute(
* PosixFilePermissions.fromString("rwxrwx---")))
```
This has something to do with yarn setting a umask or setgid to something
when it starts the container process. I don't remember the details because I
tried this early last year.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
