[
https://issues.apache.org/jira/browse/YARN-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898331#comment-15898331
]
Wangda Tan commented on YARN-6285:
----------------------------------
[~zhaoyunjiong], from your previous comment:
bq. 1. Slowness in getApplications, below stack trace files shows it spend at
least 2.25 seconds in getApplications.
Does it include ResourceRequest? From our experiences, the there're lots of
information in ResourceRequest/getLogAggregationReportsForApp and it should
contribute to most of the data/size of get_applications responses. If it is
possible, could you do a test to see, how much time will be spent if we don't
include ResourceRequest/getLogAggregationReportsForApp in the get_applications
response? This will be very helpful for us to make decisions.
To solve the problem, instead of adding the limit of apps in the server side, I
would prefer to optimize following items:
1) Add parameter to indicate if we should include
ResourceRequest/getLogAggregationReportsForApp in the response, default is true
to make it compatible. (Can be done if above experimental shows it really
helps).
2) If required, use cache to store finished apps since their report will not
change. (Can be done separately)
In addition, following items can be optimized in client side:
3) Properly use filter to include only app with expected states
(RUNNING/ACCEPT) to avoid including all apps.
4) Set limit when requesting app reports from client.
Thoughts?
> Add option to set max limit on ResourceManager for
> ApplicationClientProtocol.getApplications
> --------------------------------------------------------------------------------------------
>
> Key: YARN-6285
> URL: https://issues.apache.org/jira/browse/YARN-6285
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: yunjiong zhao
> Assignee: yunjiong zhao
> Attachments: YARN-6285.001.patch, YARN-6285.002.patch,
> YARN-6285.003.patch
>
>
> When users called ApplicationClientProtocol.getApplications, it will return
> lots of data, and generate lots of garbage on ResourceManager which caused
> long time GC.
> For example, on one of our RM, when called rest API " http://<rm http
> address:port>/ws/v1/cluster/apps" it can return 150MB data which have 944
> applications.
> getApplications have limit parameter, but some user might not set it, and
> then the limit will be Long.MAX_VALUE.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
