[
https://issues.apache.org/jira/browse/YARN-11285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-11285:
---------------------------------
Attachment: TestConcurrency.java
> LocalizedResources are leaked and its LocalPath are not cleared
> ---------------------------------------------------------------
>
> Key: YARN-11285
> URL: https://issues.apache.org/jira/browse/YARN-11285
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 3.2.1
> Reporter: Prabhu Joseph
> Assignee: Prabhu Joseph
> Priority: Major
> Attachments: TestConcurrency.java
>
>
> LocalizedResources are leaked and its LocalPath are not cleared from NM Local
> Directories. When multiple containers are initialized at same time,
> LocalResourcesTrackerImpl REQUEST handler could create and handle multiple
> LocalizedResource object for the same input path due to race condition in
> [below
> code|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTrackerImpl.java#L149]
> {code}
> case REQUEST:
> LocalResourceRequest req = event.getLocalResourceRequest();
> LocalizedResource rsrc = localrsrc.get(req);
> ....
> if (null == rsrc) {
> rsrc = new LocalizedResource(req, dispatcher);
> localrsrc.put(req, rsrc);
> }
> rsrc.handle(event);
> {code}
> Each container will have separate LocalizedResource object and separate local
> path like below.
> {code}
> /mnt/yarn/usercache/hive/filecache/6/2552419:
> total 28456
> -r-x------ 1 yarn yarn 29135164 Aug 7 10:24
> hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
> /mnt/yarn/usercache/hive/filecache/6/2552420:
> total 28456
> -r-x------ 1 yarn yarn 29135164 Aug 7 10:24
> hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
> /mnt/yarn/usercache/hive/filecache/6/2552421:
> total 28456
> -r-x------ 1 yarn yarn 29135164 Aug 7 10:24
> hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
> /mnt/yarn/usercache/hive/filecache/6/2552422:
> total 28456
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
