Github user suyanNone commented on the pull request:
https://github.com/apache/spark/pull/12735#issuecomment-215294953
```
registeredExecutorFile =
findRegisteredExecutorFile(conf.getTrimmedStrings("yarn.nodemanager.local-dirs"));
```
we got that local dirs for yarnConfig, there a lot of dirs, but current
we always adopt the first dir. right?
first assume there don't exist any meta file:
if the first disk had been removed, or had disk errors, like read-only
filesystem/input/output errors/no space left. it will cause
ExternalShuffleBlockResolver to create a new leveldb file, but it will
failed...and throw IOException, and this IOException will be engulfed by
YarnShufflerService
```
try {
blockHandler = new ExternalShuffleBlockHandler(transportConf,
registeredExecutorFile);
} catch (Exception e) {
logger.error("Failed to initialize external shuffle service", e);
}
```
So may be we can choose a more healthy disk dir for storing meta file, to
avoid necessary exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
