[
https://issues.apache.org/jira/browse/YARN-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156079#comment-14156079
]
Anubhav Dhoot commented on YARN-2624:
-------------------------------------
The fix addresses the scenario moving from pre node manager recovery to
turning on node manager recovery. As per YARN-1338 the directories are not
cleaned up inorder to preserve running containers. But uniqueNumberGenerator
will not know about preexisting directories which were normally deleted on NM
startup and are unknown to recovery enabled NM. In this case we still want
directory cleanup to happen.
> Resource Localization fails on a cluster due to existing cache directories
> --------------------------------------------------------------------------
>
> Key: YARN-2624
> URL: https://issues.apache.org/jira/browse/YARN-2624
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.5.1
> Reporter: Anubhav Dhoot
> Assignee: Anubhav Dhoot
> Priority: Blocker
> Attachments: YARN-2624.001.patch, YARN-2624.001.patch
>
>
> We have found resource localization fails on a cluster with following error
> in certain cases.
> {noformat}
> INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Failed to download rsrc { {
> hdfs://<blahhostname>:8020/tmp/hive-hive/hive_2014-09-29_14-55-45_184_6531377394813896912-12/-mr-10004/95a07b90-2448-48fc-bcda-cdb7400b4975/map.xml,
> 1412027745352, FILE, null
> },pending,[(container_1411670948067_0009_02_000001)],443533288192637,DOWNLOADING}
> java.io.IOException: Rename cannot overwrite non empty destination directory
> /data/yarn/nm/filecache/27
> at
> org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:716)
> at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:228)
> at
> org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:659)
> at org.apache.hadoop.fs.FileContext.rename(FileContext.java:906)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:366)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:59)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
