[ 
https://issues.apache.org/jira/browse/YARN-2902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388937#comment-14388937
 ] 

Sangjin Lee commented on YARN-2902:
-----------------------------------

Sorry it took me a while to get to this. Here is an excerpt of the log when 
this happened:

{noformat}
2015-03-05 00:20:17,529 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
 Application application_1418357586203_2035414 transitioned from INITING to 
RUNNING
2015-03-05 00:20:17,532 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: 
Container container_1418357586203_2035414_01_000486 transitioned from NEW to 
LOCALIZING
2015-03-05 00:20:17,532 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got 
event CONTAINER_INIT for appId application_1418357586203_2035414
2015-03-05 00:20:17,532 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got 
event APPLICATION_INIT for appId application_1418357586203_2035414
2015-03-05 00:20:17,532 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got 
APPLICATION_INIT for service mapreduce_shuffle
2015-03-05 00:20:17,532 INFO org.apache.hadoop.mapred.ShuffleHandler: Added 
token for job_1418357586203_2035414

2015-03-05 00:20:17,532 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
 Resource 
hdfs://hadoop-cluster/sharedcache/3/7/9/37904df39b3fa3ad1e23451e3c2ca718caf148b58b214b3daa2b14ea5a17277b/foo.jar
 transitioned from INIT to DOWNLOADING
2015-03-05 00:20:17,537 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
 Downloading public rsrc:{ 
hdfs://hadoop-cluster/sharedcache/3/7/9/37904df39b3fa3ad1e23451e3c2ca718caf148b58b214b3daa2b14ea5a17277b/foo.jar,
 1425430654133, FILE, null }

2015-03-05 00:28:44,388 INFO 
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=alice  
IP=10.53.186.122        OPERATION=Stop Container Request        
TARGET=ContainerManageImpl      RESULT=SUCCESS  
APPID=application_1418357586203_2035414 
CONTAINERID=container_1418357586203_2035414_01_000486

2015-03-05 00:30:51,731 ERROR 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl:
 Attempt to remove resource: { { 
hdfs://hadoop-cluster/sharedcache/3/7/9/37904df39b3fa3ad1e23451e3c2ca718caf148b58b214b3daa2b14ea5a17277b/foo.jar,
 1425430654133, FILE, null },pending,[],1895876473278238,DOWNLOADING} with 
non-zero refcount
{noformat}

Around this time there are many public resources in the downloading state that 
generated this error. I do think these public resources were truly in the 
middle of being downloaded (yes this can happen). What's not clear to me is 
whether the trigger was the public localization timing out or the stopContainer 
request (see in the log). Sorry there is not much more information I can glean 
from the log. Let me know if you have more questions.

FYI, we're running basically 2.4.0+.

> Killing a container that is localizing can orphan resources in the 
> DOWNLOADING state
> ------------------------------------------------------------------------------------
>
>                 Key: YARN-2902
>                 URL: https://issues.apache.org/jira/browse/YARN-2902
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>    Affects Versions: 2.5.0
>            Reporter: Jason Lowe
>            Assignee: Varun Saxena
>         Attachments: YARN-2902.002.patch, YARN-2902.patch
>
>
> If a container is in the process of localizing when it is stopped/killed then 
> resources are left in the DOWNLOADING state.  If no other container comes 
> along and requests these resources they linger around with no reference 
> counts but aren't cleaned up during normal cache cleanup scans since it will 
> never delete resources in the DOWNLOADING state even if their reference count 
> is zero.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to