Prabhu Joseph created YARN-11285:
------------------------------------
Summary: LocalizedResources are leaked and its LocalPath are not
cleared
Key: YARN-11285
URL: https://issues.apache.org/jira/browse/YARN-11285
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.2.1
Reporter: Prabhu Joseph
Assignee: Prabhu Joseph
LocalizedResources are leaked and its LocalPath are not cleared from NM Local
Directories. When multiple containers are initialized at same time,
LocalResourcesTrackerImpl REQUEST handler could create and handle multiple
LocalizedResource object for the same input path due to race condition in
[below
code|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTrackerImpl.java#L149]
{code}
case REQUEST:
LocalResourceRequest req = event.getLocalResourceRequest();
LocalizedResource rsrc = localrsrc.get(req);
....
if (null == rsrc) {
rsrc = new LocalizedResource(req, dispatcher);
localrsrc.put(req, rsrc);
}
rsrc.handle(event);
{code}
Each container will have separate LocalizedResource object and separate local
path like below.
{code}
/mnt/yarn/usercache/hive/filecache/6/2552419:
total 28456
-r-x------ 1 yarn yarn 29135164 Aug 7 10:24
hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
/mnt/yarn/usercache/hive/filecache/6/2552420:
total 28456
-r-x------ 1 yarn yarn 29135164 Aug 7 10:24
hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
/mnt/yarn/usercache/hive/filecache/6/2552421:
total 28456
-r-x------ 1 yarn yarn 29135164 Aug 7 10:24
hive-exec-2.3.4.50-3fd48f33b0c0b82ab431013f0fe794dfe75c31a5027567e6865cccbb49de862b.jar
/mnt/yarn/usercache/hive/filecache/6/2552422:
total 28456
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
