[
https://issues.apache.org/jira/browse/YARN-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15081600#comment-15081600
]
Mark S commented on YARN-4540:
------------------------------
In my latest test, I updated the following configuration values at 12:41PM and
restarted the App Timeline Server, NodeManager and ResourceManager, and noticed
an updated YARN cache size in around an hour. This occurred despite the
currently configured "yarn.nodemanager.localizer.cache.cleanup.interval-ms"
value.
h5. Current Values
{code}
yarn.nodemanager.localizer.cache.target-size-mb=3072
yarn.nodemanager.localizer.cache.cleanup.interval-ms=300
{code}
h5. Log snippet (45 minutes of no logs)
{code}
2016-01-04 12:41:44,917 INFO nodemanager.NodeStatusUpdaterImpl
(NodeStatusUpdaterImpl.java:registerWithRM(358)) - Notifying ContainerManager
to unblock new container-requests
2016-01-04 13:25:09,196 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(457)) - Deleting absolute path :
/hadoop/yarn/local/usercache/<username>/filecache/2438
2016-01-04 13:25:09,201 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(457)) - Deleting absolute path :
/hadoop/yarn/local/usercache/<username>/filecache/2439
2016-01-04 13:32:45,371 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(457)) - Deleting absolute path :
/hadoop/yarn/local/usercache/<username>/filecache/2440
2016-01-04 13:32:45,372 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(457)) - Deleting absolute path :
/hadoop/yarn/local/usercache/<username>/filecache/2443
2016-01-04 13:32:45,372 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(457)) - Deleting absolute path :
/hadoop/yarn/local/usercache/<username>/filecache/2441
2016-01-04 13:32:45,373 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(457)) - Deleting absolute path :
/hadoop/yarn/local/usercache/<username>/filecache/2442
{code}
h5. YARN Cache Size
{code}
[root@linux02-<username> hadoop]# date && du -m /hadoop/yarn/ | sort -nr | head
-n 20
Mon Jan 4 13:11:53 EST 2016
3753 /hadoop/yarn/
3752 /hadoop/yarn/local/usercache/<username>/filecache
3752 /hadoop/yarn/local/usercache/<username>
3752 /hadoop/yarn/local/usercache
3752 /hadoop/yarn/local
163 /hadoop/yarn/local/usercache/<username>/filecache/2467
163 /hadoop/yarn/local/usercache/<username>/filecache/2465
163 /hadoop/yarn/local/usercache/<username>/filecache/2463
163 /hadoop/yarn/local/usercache/<username>/filecache/2461
163 /hadoop/yarn/local/usercache/<username>/filecache/2459
163 /hadoop/yarn/local/usercache/<username>/filecache/2457
163 /hadoop/yarn/local/usercache/<username>/filecache/2455
163 /hadoop/yarn/local/usercache/<username>/filecache/2453
163 /hadoop/yarn/local/usercache/<username>/filecache/2451
163 /hadoop/yarn/local/usercache/<username>/filecache/2449
163 /hadoop/yarn/local/usercache/<username>/filecache/2447
163 /hadoop/yarn/local/usercache/<username>/filecache/2445
163 /hadoop/yarn/local/usercache/<username>/filecache/2443
163 /hadoop/yarn/local/usercache/<username>/filecache/2441
163 /hadoop/yarn/local/usercache/<username>/filecache/2439
[root@linux02-<username> hadoop]#
[root@linux02-<username> hadoop]#
[root@linux02-<username> hadoop]# date && du -m /hadoop/yarn/ | sort -nr | head
-n 20
Mon Jan 4 13:54:11 EST 2016
3002 /hadoop/yarn/local/usercache/<username>/filecache
3002 /hadoop/yarn/local/usercache/<username>
3002 /hadoop/yarn/local/usercache
3002 /hadoop/yarn/local
3002 /hadoop/yarn/
163 /hadoop/yarn/local/usercache/<username>/filecache/2467
163 /hadoop/yarn/local/usercache/<username>/filecache/2465
163 /hadoop/yarn/local/usercache/<username>/filecache/2463
163 /hadoop/yarn/local/usercache/<username>/filecache/2461
163 /hadoop/yarn/local/usercache/<username>/filecache/2459
163 /hadoop/yarn/local/usercache/<username>/filecache/2457
163 /hadoop/yarn/local/usercache/<username>/filecache/2455
163 /hadoop/yarn/local/usercache/<username>/filecache/2453
163 /hadoop/yarn/local/usercache/<username>/filecache/2451
163 /hadoop/yarn/local/usercache/<username>/filecache/2449
163 /hadoop/yarn/local/usercache/<username>/filecache/2447
163 /hadoop/yarn/local/usercache/<username>/filecache/2445
88 /hadoop/yarn/local/usercache/<username>/filecache/2466
88 /hadoop/yarn/local/usercache/<username>/filecache/2464
88 /hadoop/yarn/local/usercache/<username>/filecache/2462
{code}
> Yarn property "yarn.nodemanager.localizer.cache.cleanup.interval-ms" not
> working as expected
> --------------------------------------------------------------------------------------------
>
> Key: YARN-4540
> URL: https://issues.apache.org/jira/browse/YARN-4540
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager, yarn
> Affects Versions: 2.7.1
> Environment: Ambari – Version 2.1.2.1
> YARN - Version 2.7.1.2.3
> HDP - Version 2.3.2.0
> Reporter: Mark S
> Priority: Minor
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> I manually add specific YARN configuration to reduce my YARN cache size,
> however it does not get updated for the interval specified by:
> {code}
> yarn.nodemanager.localizer.cache.cleanup.interval-ms
> {code}
> I did notice that eventually my server did reduce my YARN cache size over the
> weekend.
> h5. Set Values (Should update size in 1 minute):
> {code}
> yarn.nodemanager.localizer.cache.target-size-mb=4096
> yarn.nodemanager.localizer.cache.cleanup.interval-ms=60000
> {code}
> h5. Default values (Should update size in 10 minutes):
> {code}
> yarn.nodemanager.localizer.cache.target-size-mb=10240
> yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000
> {code}
> h5. Confirming YARN cache size
> {code}
> date && du -m /hadoop/yarn | sort -nr | head -n 20
> #date && du -m . | sort -nr | head -n 20
> {code}
> h5. See also:
> *
> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
> * http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
