[
https://issues.apache.org/jira/browse/YARN-3468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14486285#comment-14486285
]
Siqi Li commented on YARN-3468:
-------------------------------
The renaming is causing a enormous number of _DEL_<timestamp> to be created,
when a NM is flapping and yarn.nodemanager.delete.debug-delay-sec is set to
several minutes.
This leads to NM full GC problem on starting up.
Here is how things go south, on NM start up it will do 3 renames, and 3 mkdir
usercache --> usercache_DEL_<timestamp>
filecache --> filecache_DEL_<timestamp>
nmPrivate --> nmPrivate_DEL_<timestamp>
mkdir usercache
mkdir filecache
mkdir nmPrivate
then it will scan all DEL dirs and add them to the deletion service. However,
with yarn.nodemanager.delete.debug-delay-sec set to 10 minute for example,
those DEL dirs will not be deleted immediately, but after 10 minutes.
When a NM flaps, it restarts every ~4 sec. There is not enough time(i.e. 10
minutes) for it to do the deletion. That's why more and more DEL dirs are
created, but none were removed.
As a result, /data/disk*/yarn/local/ will have hundreds of thousands of DEL
dirs, and scanning them and adding them to deletion service will blow out the
memory.
My suggestion is to prevent NMs from renaming those directories if they are
empty.
> NM should not blindly rename usercache/filecache/nmPrivate on restart
> ---------------------------------------------------------------------
>
> Key: YARN-3468
> URL: https://issues.apache.org/jira/browse/YARN-3468
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Siqi Li
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
