[ 
https://issues.apache.org/jira/browse/YARN-3832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14593497#comment-14593497
 ] 

Brahma Reddy Battula commented on YARN-3832:
--------------------------------------------

[~jlowe] Thanks for looking into this issue..
{quote}Can you look back in the NM logs to see when 
/opt/hdfsdata/HA/nmlocal/usercache/root/filecache/39 was originally 
created{quote}

I looked into the logs, it was created three days back.And NM was restarted 
today (days doent matter anyway,just for reference).And disk is not bad and not 
full.

{noformat}
2015-06-16 07:12:05,886 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
 Resource 
hdfs://hacluster/tmp/hadoop-yarn/staging/root/.staging/job_1434452428753_0004/libjars/netty-all-4.0.23.Final.jar(->/opt/hdfsdata/HA/nmlocal/usercache/root/filecache/39/netty-all-4.0.23.Final.jar)
 transitioned from DOWNLOADING to LOCALIZED
{noformat}

While stopping the NM, it thrown the following Error..HADOOP-11878 raised for 
same..

{noformat}
2015-06-19 03:09:10,528 ERROR 
org.apache.hadoop.yarn.server.nodemanager.DeletionService: Exception during 
execution of task in DeletionService
java.lang.NullPointerException
        at 
org.apache.hadoop.fs.FileContext.fixRelativePart(FileContext.java:274)
        at org.apache.hadoop.fs.FileContext.delete(FileContext.java:761)
        at 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.deleteAsUser(DefaultContainerExecutor.java:458)
        at 
org.apache.hadoop.yarn.server.nodemanager.DeletionService$FileDeletionTask.run(DeletionService.java:293)
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
{noformat}

AFAIK statestore recreated after starting ..Hence old state store is out of 
sync or problem with deleting the cache entries

> Resource Localization fails on a cluster due to existing cache directories
> --------------------------------------------------------------------------
>
>                 Key: YARN-3832
>                 URL: https://issues.apache.org/jira/browse/YARN-3832
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.7.0
>            Reporter: Ranga Swamy
>            Assignee: Brahma Reddy Battula
>
>  *We have found resource localization fails on a cluster with following 
> error.* 
>  
> Got this error in hadoop-2.7.0 release which was fixed in 2.6.0 (YARN-2624)
> {noformat}
> Application application_1434703279149_0057 failed 2 times due to AM Container 
> for appattempt_1434703279149_0057_000002 exited with exitCode: -1000
> For more detailed output, check application tracking 
> page:http://S0559LDPag68:45020/cluster/app/application_1434703279149_0057Then,
>  click on links to logs of each attempt.
> Diagnostics: Rename cannot overwrite non empty destination directory 
> /opt/hdfsdata/HA/nmlocal/usercache/root/filecache/39
> java.io.IOException: Rename cannot overwrite non empty destination directory 
> /opt/hdfsdata/HA/nmlocal/usercache/root/filecache/39
> at 
> org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:735)
> at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:244)
> at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:678)
> at org.apache.hadoop.fs.FileContext.rename(FileContext.java:958)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:366)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Failing this attempt. Failing the application.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to