[
https://issues.apache.org/jira/browse/YARN-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15092234#comment-15092234
]
Mark S commented on YARN-4540:
------------------------------
Just to follow up. The "/hadoop/yarn/local/usercache/<username>/filecache" is
a total of 3752 MB in size. Also the command shown above "date && du -m
/hadoop/yarn/ | sort -nr | head -n 20" is specifically showing only the top 20
values ("head -n 20"), so there is a whole bunch of filecache sizes not shown.
As such, the calculation mentioning "163 * 15" is only the sum of a partial set.
Just a quick update to try and clear up any misunderstanding. I was NOT
running any YARN jobs (My Hadoop cluster is for testing and was idle), and as
such the directory "/hadoop/yarn/local" was at a stable constant size.
Initially my "/hadoop/yarn/local" size was around 10 GB, and then I updated
"yarn.nodemanager.localizer.cache.target-size-mb" to a much smaller value
(Think around half). Then I ran the test again when the "/hadoop/yarn/local"
size was around 5GB and set the
"yarn.nodemanager.localizer.cache.target-size-mb" property to about 4GB. Then
I ran another test. My "/hadoop/yarn/local" size was around 4GB and I set the
"yarn.nodemanager.localizer.cache.target-size-mb" property to about 3 GB.
> Yarn property "yarn.nodemanager.localizer.cache.cleanup.interval-ms" not
> working as expected
> --------------------------------------------------------------------------------------------
>
> Key: YARN-4540
> URL: https://issues.apache.org/jira/browse/YARN-4540
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager, yarn
> Affects Versions: 2.7.1
> Environment: Ambari – Version 2.1.2.1
> YARN - Version 2.7.1.2.3
> HDP - Version 2.3.2.0
> Reporter: Mark S
> Priority: Minor
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> I manually add specific YARN configuration to reduce my YARN cache size,
> however it does not get updated for the interval specified by:
> {code}
> yarn.nodemanager.localizer.cache.cleanup.interval-ms
> {code}
> I did notice that eventually my server did reduce my YARN cache size over the
> weekend.
> h5. Set Values (Should update size in 1 minute):
> {code}
> yarn.nodemanager.localizer.cache.target-size-mb=4096
> yarn.nodemanager.localizer.cache.cleanup.interval-ms=60000
> {code}
> h5. Default values (Should update size in 10 minutes):
> {code}
> yarn.nodemanager.localizer.cache.target-size-mb=10240
> yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000
> {code}
> h5. Confirming YARN cache size
> {code}
> date && du -m /hadoop/yarn | sort -nr | head -n 20
> #date && du -m . | sort -nr | head -n 20
> {code}
> h5. See also:
> *
> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
> * http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
