[
https://issues.apache.org/jira/browse/YARN-11285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-11285:
---------------------------------
Description:
LocalizedResources are leaked and its LocalPath are not cleared from NM Local
Directories.
Each container has separate LocalizedResource object and separate local path
like below.
{code}
/mnt/yarn/usercache/hive/filecache/6/2552419:
total 28456
-r-x------ 1 yarn yarn 29135164 Aug 7 10:24
hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
/mnt/yarn/usercache/hive/filecache/6/2552420:
total 28456
-r-x------ 1 yarn yarn 29135164 Aug 7 10:24
hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
/mnt/yarn/usercache/hive/filecache/6/2552421:
total 28456
-r-x------ 1 yarn yarn 29135164 Aug 7 10:24
hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
/mnt/yarn/usercache/hive/filecache/6/2552422:
total 28456
{code}
was:
LocalizedResources are leaked and its LocalPath are not cleared from NM Local
Directories. When multiple containers are initialized at same time,
LocalResourcesTrackerImpl REQUEST handler could create and handle multiple
LocalizedResource object for the same input path due to race condition in
[below
code|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTrackerImpl.java#L149]
{code}
case REQUEST:
LocalResourceRequest req = event.getLocalResourceRequest();
LocalizedResource rsrc = localrsrc.get(req);
....
if (null == rsrc) {
rsrc = new LocalizedResource(req, dispatcher);
localrsrc.put(req, rsrc);
}
rsrc.handle(event);
{code}
Each container will have separate LocalizedResource object and separate local
path like below.
{code}
/mnt/yarn/usercache/hive/filecache/6/2552419:
total 28456
-r-x------ 1 yarn yarn 29135164 Aug 7 10:24
hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
/mnt/yarn/usercache/hive/filecache/6/2552420:
total 28456
-r-x------ 1 yarn yarn 29135164 Aug 7 10:24
hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
/mnt/yarn/usercache/hive/filecache/6/2552421:
total 28456
-r-x------ 1 yarn yarn 29135164 Aug 7 10:24
hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
/mnt/yarn/usercache/hive/filecache/6/2552422:
total 28456
{code}
> LocalizedResources are leaked and its LocalPath are not cleared
> ---------------------------------------------------------------
>
> Key: YARN-11285
> URL: https://issues.apache.org/jira/browse/YARN-11285
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 3.2.1
> Reporter: Prabhu Joseph
> Assignee: Prabhu Joseph
> Priority: Major
>
> LocalizedResources are leaked and its LocalPath are not cleared from NM Local
> Directories.
> Each container has separate LocalizedResource object and separate local path
> like below.
> {code}
> /mnt/yarn/usercache/hive/filecache/6/2552419:
> total 28456
> -r-x------ 1 yarn yarn 29135164 Aug 7 10:24
> hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
> /mnt/yarn/usercache/hive/filecache/6/2552420:
> total 28456
> -r-x------ 1 yarn yarn 29135164 Aug 7 10:24
> hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
> /mnt/yarn/usercache/hive/filecache/6/2552421:
> total 28456
> -r-x------ 1 yarn yarn 29135164 Aug 7 10:24
> hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
> /mnt/yarn/usercache/hive/filecache/6/2552422:
> total 28456
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
