[
https://issues.apache.org/jira/browse/YARN-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
agoodboy updated YARN-9778:
---------------------------
Issue Type: Bug (was: Improvement)
> Nodemanager does not clean public cache(filecache)
> --------------------------------------------------
>
> Key: YARN-9778
> URL: https://issues.apache.org/jira/browse/YARN-9778
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Affects Versions: 3.1.2
> Environment: HDP 3.1.0.78
> Reporter: agoodboy
> Priority: Major
>
> Nodemanager does not clean local filecache dir event the size exceeds the
> config in yarn-site. The config in yarn-site.xml is as follows:
> <property>
> <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>
> <value>600000</value>
> </property>
> <property>
> <name>yarn.nodemanager.localizer.cache.target-size-mb</name>
> <value>10240</value>
> </property>
> <property>
> <name>yarn.nodemanager.container-executor.class</name>
>
> <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
> </property>
>
> I use docker to run my program and in docker container I will download file
> from hdfs to local dir. But after docker container killed or exit, the files
> doesn't cleaned by nodemanager, hence, the filecache dir increases and node
> enters unhealthy state. The docker start command with a mounted dir like this:
> -v=/data1/hadoop/yarn/local/filecache/2115/models.tar.gz/models:/home/hadoop/xdl/models:rw
>
> -v=/data1/hadoop/yarn/local/filecache/2116:/data1/hadoop/yarn/local/filecache/2116
>
> -v=/data1/hadoop/yarn/local/filecache/2117:/data1/hadoop/yarn/local/filecache/2117.
>
> For example, the filecache dir size is
> $ sudo du -sh .filecache$ sudo du -sh .112G .
>
> But nodemanager still does not clean it event I set cache size is 10GB.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
