zhihai xu commented on YARN-3591:

[~lavkesh], thanks for the new patch. It looks like your new patch will also 
call removeResource on DirectoryCollection.fullDirs. Most likely the files in 
fullDirs can still be used, Dir in fullDirs may become good after the files in 
it are deleted by CacheCleanup. If a localized resource is in fullDirs, reusing 
it for same LocalResourceRequest will be better than removing it. Another 
problem is these files are still at the disks, when the NM restart, we will hit 
the issue YARN-2624. LocalResourcesTrackerImpl#getPathForLocalization may 
allocate same name directory, which cause localization failure. This issue 
looks like much more complicated than we thought.
IMHO, we can add two parameters in onDirsChanged: dirs(newErrorDirs) which are 
changed from localDirs and fullDirs to errorDirs , dirs(newRepairedDirs) which 
are changed from errorDirs to localDirs or fullDirs. We can call removeResource 
for the localized resources in newErrorDirs.
We can call cleanUpLocalDir to delete the obsolete files in newRepairedDirs. 
With this change, we may solve your previous concern "What about zombie files 
lying in the various paths". Also we should save the errorDirs in StateStore 
for NM recovery, so we can delete the obsolete files in these errorDirs after  
NM restart.

> Resource Localisation on a bad disk causes subsequent containers failure 
> -------------------------------------------------------------------------
>                 Key: YARN-3591
>                 URL: https://issues.apache.org/jira/browse/YARN-3591
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.7.0
>            Reporter: Lavkesh Lahngir
>            Assignee: Lavkesh Lahngir
>         Attachments: 0001-YARN-3591.1.patch, 0001-YARN-3591.patch, 
> YARN-3591.2.patch, YARN-3591.3.patch, YARN-3591.4.patch
> It happens when a resource is localised on the disk, after localising that 
> disk has gone bad. NM keeps paths for localised resources in memory.  At the 
> time of resource request isResourcePresent(rsrc) will be called which calls 
> file.exists() on the localised path.
> In some cases when disk has gone bad, inodes are stilled cached and 
> file.exists() returns true. But at the time of reading, file will not open.
> Note: file.exists() actually calls stat64 natively which returns true because 
> it was able to find inode information from the OS.
> A proposal is to call file.list() on the parent path of the resource, which 
> will call open() natively. If the disk is good it should return an array of 
> paths with length at-least 1.

This message was sent by Atlassian JIRA

Reply via email to