Siqi Li commented on YARN-3468:

The renaming is causing a enormous number of _DEL_<timestamp> to be created, 
when a NM is flapping and yarn.nodemanager.delete.debug-delay-sec is set to 
several minutes.

This leads to NM full GC problem on starting up.

Here is how things go south, on NM start up it will do 3 renames, and 3 mkdir

usercache --> usercache_DEL_<timestamp>
filecache   --> filecache_DEL_<timestamp>
nmPrivate --> nmPrivate_DEL_<timestamp>

mkdir usercache
mkdir filecache
mkdir nmPrivate

then it will scan all DEL dirs and add them to the deletion service. However, 
with yarn.nodemanager.delete.debug-delay-sec set to 10 minute for example, 
those DEL dirs will not be deleted immediately, but after 10 minutes.

When a NM flaps, it restarts every ~4 sec. There is not enough time(i.e. 10 
minutes) for it to do the deletion. That's why more and more DEL dirs are 
created, but none were removed.

As a result, /data/disk*/yarn/local/ will have hundreds of thousands of DEL 
dirs, and scanning them and adding them to deletion service will blow out the 

My suggestion is to prevent NMs from renaming those directories if they are 

> NM should not blindly rename usercache/filecache/nmPrivate on restart
> ---------------------------------------------------------------------
>                 Key: YARN-3468
>                 URL: https://issues.apache.org/jira/browse/YARN-3468
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Siqi Li

This message was sent by Atlassian JIRA

Reply via email to