[ 
https://issues.apache.org/jira/browse/YARN-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14390305#comment-14390305
 ] 

vishal.rajan commented on YARN-2624:
------------------------------------

seems like this issue still persist in yarn  2.6.0 under certain conditions.

Dump of the log relating to this issue.
================================
15/04/01 12:13:20 ERROR test.Job: Task error: Rename cannot overwrite non empty 
destination directory /grid/6/yarn/local/usercache/azkaban/filecache/344860
java.io.IOException: Rename cannot overwrite non empty destination directory 
/grid/6/yarn/local/usercache/azkaban/filecache/344860
        at 
org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:716)
        at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:228)
        at 
org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:659)
        at org.apache.hadoop.fs.FileContext.rename(FileContext.java:909)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:364)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
=============================
yarn version : hadoop-2-2-0-0-2041-yarn                   2.6.0.2.2.0.0-2041
=============================

This node was taken OOR for maintanance, and when it was added back to the 
cluster, seems like this 344860 directory was not removed before assigning it 
to the new container.




> Resource Localization fails on a cluster due to existing cache directories
> --------------------------------------------------------------------------
>
>                 Key: YARN-2624
>                 URL: https://issues.apache.org/jira/browse/YARN-2624
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.5.1
>            Reporter: Anubhav Dhoot
>            Assignee: Anubhav Dhoot
>            Priority: Blocker
>             Fix For: 2.6.0
>
>         Attachments: YARN-2624.001.patch, YARN-2624.001.patch
>
>
> We have found resource localization fails on a cluster with following error 
> in certain cases.
> {noformat}
> INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>  Failed to download rsrc { { 
> hdfs://<blahhostname>:8020/tmp/hive-hive/hive_2014-09-29_14-55-45_184_6531377394813896912-12/-mr-10004/95a07b90-2448-48fc-bcda-cdb7400b4975/map.xml,
>  1412027745352, FILE, null 
> },pending,[(container_1411670948067_0009_02_000001)],443533288192637,DOWNLOADING}
> java.io.IOException: Rename cannot overwrite non empty destination directory 
> /data/yarn/nm/filecache/27
>       at 
> org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:716)
>       at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:228)
>       at 
> org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:659)
>       at org.apache.hadoop.fs.FileContext.rename(FileContext.java:906)
>       at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:366)
>       at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:59)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to