[jira] [Commented] (YARN-4540) Yarn property "yarn.nodemanager.localizer.cache.cleanup.interval-ms" not working as expected

2016-01-11 Thread Mark S (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093159#comment-15093159
 ] 

Mark S commented on YARN-4540:
--

In response to your first two questions.  Basically I ran a lot of YARN jobs 
and noticed that the "/hadoop/yarn/local/usercache/" directory grew 
to about 10 GB.  I then stopped all my YARN jobs.  Then I attempted to reduce 
the "/hadoop/yarn/local/usercache/" directory size by setting the 
"yarn.nodemanager.localizer.cache.target-size-mb" property and waiting.

In response to your last question.  Yes, it seemed that the caches where 
deleted to the target size, but in a much greater time interval then specified 
in the "yarn.nodemanger.localizer.cache.cleanup.interval-ms" property.



> Yarn property "yarn.nodemanager.localizer.cache.cleanup.interval-ms" not 
> working as expected
> 
>
> Key: YARN-4540
> URL: https://issues.apache.org/jira/browse/YARN-4540
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager, yarn
>Affects Versions: 2.7.1
> Environment: Ambari – Version 2.1.2.1
> YARN - Version 2.7.1.2.3
> HDP - Version 2.3.2.0
>Reporter: Mark S
>Priority: Minor
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> I manually add specific YARN configuration to reduce my YARN cache size, 
> however it does not get updated for the interval specified by:
> {code}
> yarn.nodemanager.localizer.cache.cleanup.interval-ms
> {code}
> I did notice that eventually my server did reduce my YARN cache size over the 
> weekend.
> h5. Set Values (Should update size in 1 minute):
> {code}
> yarn.nodemanager.localizer.cache.target-size-mb=4096
> yarn.nodemanager.localizer.cache.cleanup.interval-ms=6
> {code}
> h5. Default values (Should update size in 10 minutes):
> {code}
> yarn.nodemanager.localizer.cache.target-size-mb=10240
> yarn.nodemanager.localizer.cache.cleanup.interval-ms=60
> {code}
> h5.  Confirming YARN cache size
> {code}
> date && du -m /hadoop/yarn | sort -nr | head -n 20
> #date && du -m . | sort -nr | head -n 20
> {code}
> h5. See also:
> * 
> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
> * http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-4540) Yarn property "yarn.nodemanager.localizer.cache.cleanup.interval-ms" not working as expected

2016-01-11 Thread Mark S (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092234#comment-15092234
 ] 

Mark S commented on YARN-4540:
--

Just to follow up.  The "/hadoop/yarn/local/usercache//filecache" is 
a total of 3752 MB in size.  Also the command shown above "date && du -m 
/hadoop/yarn/ | sort -nr | head -n 20" is specifically showing only the top 20 
values ("head -n 20"), so there is a whole bunch of filecache sizes not shown. 
As such, the calculation mentioning "163 * 15" is only the sum of a partial set.

Just a quick update to try and clear up any misunderstanding.  I was NOT 
running any YARN jobs (My Hadoop cluster is for testing and was idle), and as 
such the directory "/hadoop/yarn/local" was at a stable constant size. 
Initially my "/hadoop/yarn/local" size was around 10 GB, and then I updated 
"yarn.nodemanager.localizer.cache.target-size-mb" to a much smaller value 
(Think around half).  Then I ran the test again when the "/hadoop/yarn/local" 
size was around 5GB and set the 
"yarn.nodemanager.localizer.cache.target-size-mb" property to about 4GB.  Then 
I ran another test.  My "/hadoop/yarn/local" size was around 4GB and I set the 
"yarn.nodemanager.localizer.cache.target-size-mb" property to about 3 GB.

> Yarn property "yarn.nodemanager.localizer.cache.cleanup.interval-ms" not 
> working as expected
> 
>
> Key: YARN-4540
> URL: https://issues.apache.org/jira/browse/YARN-4540
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager, yarn
>Affects Versions: 2.7.1
> Environment: Ambari – Version 2.1.2.1
> YARN - Version 2.7.1.2.3
> HDP - Version 2.3.2.0
>Reporter: Mark S
>Priority: Minor
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> I manually add specific YARN configuration to reduce my YARN cache size, 
> however it does not get updated for the interval specified by:
> {code}
> yarn.nodemanager.localizer.cache.cleanup.interval-ms
> {code}
> I did notice that eventually my server did reduce my YARN cache size over the 
> weekend.
> h5. Set Values (Should update size in 1 minute):
> {code}
> yarn.nodemanager.localizer.cache.target-size-mb=4096
> yarn.nodemanager.localizer.cache.cleanup.interval-ms=6
> {code}
> h5. Default values (Should update size in 10 minutes):
> {code}
> yarn.nodemanager.localizer.cache.target-size-mb=10240
> yarn.nodemanager.localizer.cache.cleanup.interval-ms=60
> {code}
> h5.  Confirming YARN cache size
> {code}
> date && du -m /hadoop/yarn | sort -nr | head -n 20
> #date && du -m . | sort -nr | head -n 20
> {code}
> h5. See also:
> * 
> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
> * http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (YARN-4540) Yarn property "yarn.nodemanager.localizer.cache.cleanup.interval-ms" not working as expected

2016-01-04 Thread Mark S (JIRA) 
Mark S created YARN-4540:


 Summary: Yarn property 
"yarn.nodemanager.localizer.cache.cleanup.interval-ms" not working as expected
 Key: YARN-4540
 URL: https://issues.apache.org/jira/browse/YARN-4540
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.7.1
 Environment: Ambari – Version 2.1.2.1
YARN - Version 2.7.1.2.3
HDP - Version 2.3.2.0
Reporter: Mark S
Priority: Minor


I manually add specific YARN configuration to reduce my YARN cache size, 
however it does not get updated for the interval specified by:
{code}
yarn.nodemanager.localizer.cache.cleanup.interval-ms
{code}
I did notice that eventually my server did reduce my YARN cache size over the 
weekend.


h5. Set Values (Should update size in 1 minute):
{code}
yarn.nodemanager.localizer.cache.target-size-mb=4096
yarn.nodemanager.localizer.cache.cleanup.interval-ms=6
{code}

h5. Default values (Should update size in 10 minutes):
{code}
yarn.nodemanager.localizer.cache.target-size-mb=10240
yarn.nodemanager.localizer.cache.cleanup.interval-ms=60
{code}

h5.  Confirming YARN cache size
{code}
date && du -m /hadoop/yarn | sort -nr | head -n 20
#date && du -m . | sort -nr | head -n 20
{code}

h5. See also:
* 
https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
* http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (YARN-4540) Yarn property "yarn.nodemanager.localizer.cache.cleanup.interval-ms" not working as expected

2016-01-04 Thread Mark S (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-4540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark S updated YARN-4540:
-
Component/s: yarn

> Yarn property "yarn.nodemanager.localizer.cache.cleanup.interval-ms" not 
> working as expected
> 
>
> Key: YARN-4540
> URL: https://issues.apache.org/jira/browse/YARN-4540
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager, yarn
>Affects Versions: 2.7.1
> Environment: Ambari – Version 2.1.2.1
> YARN - Version 2.7.1.2.3
> HDP - Version 2.3.2.0
>Reporter: Mark S
>Priority: Minor
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> I manually add specific YARN configuration to reduce my YARN cache size, 
> however it does not get updated for the interval specified by:
> {code}
> yarn.nodemanager.localizer.cache.cleanup.interval-ms
> {code}
> I did notice that eventually my server did reduce my YARN cache size over the 
> weekend.
> h5. Set Values (Should update size in 1 minute):
> {code}
> yarn.nodemanager.localizer.cache.target-size-mb=4096
> yarn.nodemanager.localizer.cache.cleanup.interval-ms=6
> {code}
> h5. Default values (Should update size in 10 minutes):
> {code}
> yarn.nodemanager.localizer.cache.target-size-mb=10240
> yarn.nodemanager.localizer.cache.cleanup.interval-ms=60
> {code}
> h5.  Confirming YARN cache size
> {code}
> date && du -m /hadoop/yarn | sort -nr | head -n 20
> #date && du -m . | sort -nr | head -n 20
> {code}
> h5. See also:
> * 
> https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
> * http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)