dongjoon-hyun commented on a change in pull request #25047:
[WIP][SPARK-27371][CORE] Support GPU-aware resources scheduling in Standalone
URL: https://github.com/apache/spark/pull/25047#discussion_r300217803
##########
File path: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
##########
@@ -190,19 +189,8 @@ private[deploy] class Worker(
private def createWorkDir() {
workDir = Option(workDirPath).map(new File(_)).getOrElse(new
File(sparkHome, "work"))
- try {
- // This sporadically fails - not sure why ... !workDir.exists() &&
!workDir.mkdirs()
- // So attempting to create and then check if directory was created or
not.
- workDir.mkdirs()
- if ( !workDir.exists() || !workDir.isDirectory) {
- logError("Failed to create work directory " + workDir)
- System.exit(1)
- }
- assert (workDir.isDirectory)
- } catch {
- case e: Exception =>
- logError("Failed to create work directory " + workDir, e)
- System.exit(1)
+ if (!Utils.createDirectory(workDir)) {
Review comment:
Ur, `Utils.createDirectory` works differently from `workDir.mkdirs()`.
Do we need to change the current behavior of `createWorkDir` fo this PR?
This behavior change may break 3rd party tools.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
