[
https://issues.apache.org/jira/browse/SPARK-44469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744047#comment-17744047
]
Yazhi Wang commented on SPARK-44469:
------------------------------------
I'll work on it.
> Utils.getOrCreateLocalRootDirs will never take effect after the first call
> fails, even if the exception is recovered
> --------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-44469
> URL: https://issues.apache.org/jira/browse/SPARK-44469
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.4.1
> Reporter: Yazhi Wang
> Priority: Minor
>
> {code:java}
> private[spark] def getOrCreateLocalRootDirs(conf: SparkConf): Array[String] =
> {
> if (localRootDirs == null || localRootDirs.isEmpty) {
> this.synchronized {
> if (localRootDirs == null) {
> localRootDirs = getOrCreateLocalRootDirsImpl(conf)
> }
> }
> }
> localRootDirs
> }{code}
> localRootDirs is only initialized once in the Executor/Driver life cycle. If
> it fails due to a FileSystem exception (such as a full disk) during the first
> initialization, localRootDirs will be assigned a value of None instead of
> null.
> Even if the FileSystem exception recovered, the localRootDirs won't re-apply
> from FileSystem, causing the task on the Executor to continue to fail (tasks
> relay on fetchFiles locally)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
