[
https://issues.apache.org/jira/browse/YARN-8995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16862821#comment-16862821
]
Tao Yang edited comment on YARN-8995 at 6/13/19 8:27 AM:
---------------------------------------------------------
Thanks [~zhuqi] for updating the patch.
Comments for the new patch:
* Sorry to have made a mistake in my last comment, serviceInit is a more proper
place to initialize conf, then you can remove the initial value for
detailsInterval field.
* There's no need to separate name with double "\_" for "...EVENTS__INFO...",
"...EVENTS_INFO..." is ok. The annotation "The interval thousands of queue
size" can be replaced as "The interval of queue size (in thousands)".
* For parallelStream, overhead is involved in splitting the work among several
threads and joining or merging the results, I prefer using sequential stream in
this scenario which has no I/O operations and only need to count for event
types. Moreover, we can use groupingBy API like this:
{{eventQueue.stream().collect(Collectors.groupingBy(e -> e.getType(),
Collectors.counting()))}}, instead of calling Collectors#toConcurrentMap or
Collectors#toMap.
was (Author: tao yang):
Thanks [~zhuqi] for updating the patch.
Comments for the new patch:
* Sorry to have made a mistake in my last comment, serviceInit is a more proper
place to initialize conf, then you can remove the initial value for
detailsInterval field.
* There's no need to separate name with double "_" for "...EVENTS__INFO...",
"...EVENTS_INFO..." is ok. The annotation "The interval thousands of ..." can
be replaced as "The interval of ... (in thousands)".
* For parallelStream, overhead is involved in splitting the work among several
threads and joining or merging the results, I prefer using sequential stream in
this scenario which has no I/O operations and only need to count for event
types. Moreover, we can use groupingBy API like this:
{{eventQueue.stream().collect(Collectors.groupingBy(e -> e.getType(),
Collectors.counting()))}}, instead of calling Collectors#toConcurrentMap or
Collectors#toMap.
> Log the event type of the too big AsyncDispatcher event queue size, and add
> the information to the metrics.
> ------------------------------------------------------------------------------------------------------------
>
> Key: YARN-8995
> URL: https://issues.apache.org/jira/browse/YARN-8995
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: metrics, nodemanager, resourcemanager
> Affects Versions: 3.2.0
> Reporter: zhuqi
> Assignee: zhuqi
> Priority: Major
> Attachments: YARN-8995.001.patch, YARN-8995.002.patch,
> YARN-8995.003.patch
>
>
> In our growing cluster，there are unexpected situations that cause some event
> queues to block the performance of the cluster, such as the bug of
> https://issues.apache.org/jira/browse/YARN-5262 . I think it's necessary to
> log the event type of the too big event queue size, and add the information
> to the metrics, and the threshold of queue size is a parametor which can be
> changed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
