HyukjinKwon commented on code in PR #37203:
URL: https://github.com/apache/spark/pull/37203#discussion_r928339985
##########
core/src/main/scala/org/apache/spark/util/Utils.scala:
##########
@@ -920,9 +920,9 @@ private[spark] object Utils extends Logging {
// user has access to them.
randomizeInPlace(getYarnLocalDirs(conf).split(","))
} else if (conf.getenv("SPARK_EXECUTOR_DIRS") != null) {
- conf.getenv("SPARK_EXECUTOR_DIRS").split(File.pathSeparator)
+
randomizeInPlace(conf.getenv("SPARK_EXECUTOR_DIRS").split(File.pathSeparator))
Review Comment:
Hm, quick question though. `getConfiguredLocalDirs` gets invoked multiple
times, and won't it return a different directory every time when it's called?
This `spark.local.dir` can be used to pass `spark.files` around, e.g.,
`SparkFiles.get`. Wonder if that would be affected.
If this is a K8S specific issue, I think it's best to localize the fix into
K8S.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
