[
https://issues.apache.org/jira/browse/YARN-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16104478#comment-16104478
]
Manikandan R commented on YARN-65:
----------------------------------
Based on earlier suggestions, made changes to free memory by clearing unused
variables explicitly with either null or empty values, particularly
submissioncontext object. As of now, we are clearing up
app.submissionContext.getAMContainerSpec().setTokensConf(null);
inside AddApplicationToSchedulerTransition, RMAppRecoveredTransition &
FinalTransition class in RMAppImpl.java. Created a new method to clear all
unused fields including above mentioned specific field so that it can be called
from above mentioned classes.
Attached initial patch for review.
Couple of junits test cases like
TestContainerAllocation.testLogAggregationContextPassedIntoContainerToken
TestRMWebServicesAppsModification.testGetNewApplicationAndSubmit
is failing because actual value would have been cleared and hence not matching
with the expected value. To confirm whether those fields has been used in
application report, checked the AppBlock.java and don't see any usage over
there? Please correct me if this understanding is wrong. Based on feedback, we
may need to fix the Junit test cases accordingly.
In addition, to understand the memory savings, submitted an simple yarn app to
RM (local setup) in series and tried to visualize memory usage using jconsole
during GC before and after this code changes. I am not seeing any significant
difference as may be because of app is very light and size of each field inside
submissioncontext etc.
> Reduce RM app memory footprint once app has completed
> -----------------------------------------------------
>
> Key: YARN-65
> URL: https://issues.apache.org/jira/browse/YARN-65
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
> Affects Versions: 0.23.3
> Reporter: Jason Lowe
> Assignee: Naganarasimha G R
> Attachments: YARN-65.001.patch
>
>
> The ResourceManager holds onto a configurable number of completed
> applications (yarn.resource.max-completed-applications, defaults to 10000),
> and the memory footprint of these completed applications can be significant.
> For example, the {{submissionContext}} in RMAppImpl contains references to
> protocolbuffer objects and other items that probably aren't necessary to keep
> around once the application has completed. We could significantly reduce the
> memory footprint of the RM by releasing objects that are no longer necessary
> once an application completes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
