Yazhi Wang created SPARK-44469:
----------------------------------
Summary: Utils.getOrCreateLocalRootDirs will never take effect
after the first call fails, even if the exception is recovered
Key: SPARK-44469
URL: https://issues.apache.org/jira/browse/SPARK-44469
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.4.1
Reporter: Yazhi Wang
{code:java}
private[spark] def getOrCreateLocalRootDirs(conf: SparkConf): Array[String] = {
if (localRootDirs == null || localRootDirs.isEmpty) {
this.synchronized {
if (localRootDirs == null) {
localRootDirs = getOrCreateLocalRootDirsImpl(conf)
}
}
}
localRootDirs
}{code}
localRootDirs is only initialized once in the Executor/Driver life cycle. If it
fails due to a FileSystem exception (such as a full disk) during the first
initialization, localRootDirs will be assigned a value of None instead of null.
Even if the FileSystem exception recovered, the localRootDirs won't re-apply
from FileSystem, causing the task on the Executor to continue to fail (tasks
relay on fetchFiles locally)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
