[
https://issues.apache.org/jira/browse/YARN-3591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14569727#comment-14569727
]
zhihai xu commented on YARN-3591:
---------------------------------
Hi [~lavkesh], I think we can create a separate JIRA for storing local Error
directories in NM state store, which will be a good enhancement.
thanks [~sunilg]! Adding a new API to get local error directories is also a
good suggestion. But I think it will be enough to just check newErrorDirs
instead of all errorDirs.
To better support NM recovery and make DirsChangeListener interface simple, I
propose the following changes:
1.In DirectoryCollection, notify listener when any set of dirs(localDirs,
errorDirs and fullDirs) are changed
The code change at {{DirectoryCollection#checkDirs}} looks like the following:
{code}
bool needNotifyListener = false;
needNotifyListener = setChanged;
for (String dir : preCheckFullDirs) {
if (postCheckOtherDirs.contains(dir)) {
needNotifyListener = true;
LOG.warn("Directory " + dir + " error "
+ dirsFailedCheck.get(dir).message);
}
}
for (String dir : preCheckOtherErrorDirs) {
if (postCheckFullDirs.contains(dir)) {
needNotifyListener = true;
LOG.warn("Directory " + dir + " error "
+ dirsFailedCheck.get(dir).message);
}
}
if (needNotifyListener) {
for (DirsChangeListener listener : dirsChangeListeners) {
listener.onDirsChanged();
}
}
{code}
2. add an API to get local error directories.
As [~sunilg] suggested, We can add an API {{synchronized List<String>
getErrorDirs()}} in DirectoryCollection.java
We also need add an API {{public List<String> getLocalErrorDirs()}} in
LocalDirsHandlerService.java, which will call
{{DirectoryCollection#getErrorDirs}}
3. add a field {{Set<String> preLocalErrorDirs}} in
ResourceLocalizationService.java to store previous local error directories.
{{ResourceLocalizationService#preLocalErrorDirs}} should be loaded from state
store at the beginning if we support storing local Error directories in NM
state store.
4.The following is pseudo code for {{localDirsChangeListener#onDirsChanged}}:
{code}
Set<String> curLocalErrorDirs = new
HashSet<String>(dirsHandler.getLocalErrorDirs());
List<String> newErrorDirs = new ArrayList<String>();
List<String> newRepairedDirs = new ArrayList<String>();
for (String dir : curLocalErrorDirs) {
if (!preLocalErrorDirs.contains(dir)) {
newErrorDirs.add(dir);
}
}
for (String dir : preLocalErrorDirs) {
if (!curLocalErrorDirs.contains(dir)) {
newRepairedDirs.add(dir);
}
}
for (String localDir : newRepairedDirs) {
cleanUpLocalDir(lfs, delService, localDir);
}
if (!newErrorDirs.isEmpty()) {
//As Sunil suggested, checkLocalizedResources will call removeResource on those
localized resources whose parent is present in newErrorDirs.
publicRsrc.checkLocalizedResources(newErrorDirs);
for (LocalResourcesTracker tracker : privateRsrc.values()) {
tracker.checkLocalizedResources(newErrorDirs);
}
}
if (!newErrorDirs.isEmpty() || !newRepairedDirs.isEmpty()) {
preLocalErrorDirs = curLocalErrorDirs;
stateStore.storeLocalErrorDirs(StringUtils.arrayToString(curLocalErrorDirs.toArray(new
String[0])));
}
checkAndInitializeLocalDirs();
{code}
5. It will be better to move {{verifyDirUsingMkdir(testDir)}} right after
{{DiskChecker.checkDir(testDir)}} in {{DirectoryCollection#testDirs}}, so we
can detect the error directory before detecting the full directory.
Please feel free to change or add more to my proposal.
> Resource Localisation on a bad disk causes subsequent containers failure
> -------------------------------------------------------------------------
>
> Key: YARN-3591
> URL: https://issues.apache.org/jira/browse/YARN-3591
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 2.7.0
> Reporter: Lavkesh Lahngir
> Assignee: Lavkesh Lahngir
> Attachments: 0001-YARN-3591.1.patch, 0001-YARN-3591.patch,
> YARN-3591.2.patch, YARN-3591.3.patch, YARN-3591.4.patch
>
>
> It happens when a resource is localised on the disk, after localising that
> disk has gone bad. NM keeps paths for localised resources in memory. At the
> time of resource request isResourcePresent(rsrc) will be called which calls
> file.exists() on the localised path.
> In some cases when disk has gone bad, inodes are stilled cached and
> file.exists() returns true. But at the time of reading, file will not open.
> Note: file.exists() actually calls stat64 natively which returns true because
> it was able to find inode information from the OS.
> A proposal is to call file.list() on the parent path of the resource, which
> will call open() natively. If the disk is good it should return an array of
> paths with length at-least 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
