[jira] [Commented] (YARN-402) Dispatcher warn message is too late
[
https://issues.apache.org/jira/browse/YARN-402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283150#comment-16283150
]
stefanlee commented on YARN-402:
thanks [~djp] our cluster will be very busy between 3.00 a.m. and 7.00 a.m. the
event-queue size will be 4000 to 5000, so how can i optimize this?
> Dispatcher warn message is too late
> ---
>
> Key: YARN-402
> URL: https://issues.apache.org/jira/browse/YARN-402
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Lohit Vijayarenu
>Priority: Minor
>
> AsyncDispatcher throws out Warn when capacity remaining is less than 1000
> {noformat}
> if (remCapacity < 1000) {
> LOG.warn("Very low remaining capacity in the event-queue: "
> + remCapacity);
> }
> {noformat}
> What would be useful is to warn much before that, may be half full instead of
> when queue is completely full. I see that eventQueue capacity is int value.
> So, if one warn's queue has only 1000 capacity left, then service definitely
> has serious problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-402) Dispatcher warn message is too late
[
https://issues.apache.org/jira/browse/YARN-402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395402#comment-14395402
]
Junping Du commented on YARN-402:
-
Thanks [~lohit] for reporting this issue.
I think it could be a little too allergic to give a warn when half full of the
queue. By default, the size of LinkedBlockingQueue is: Interger.MAX_VALUE which
is 2^31-1. Half full means: still ~2^30 available for use so it could be too
early.
Do we want a configurable value here? I think it could be a little overkill. If
so, we may need to pick up a more reasonable fixed value here.
IMO, rmDispatcher could be the most busy AsynDispatcher in YARN today,
RMNodeEvent, SchedulerEvent, RMAppEvent, RMAppAttemptEvent,
NodeListManagerEvent, AMLauncherEvent, etc. are all get broadcasted on this
single dispatcher. Within these events, SchedulerEvent seems to be the most
active events: let's assume thousands of nodes events and thousands of
application attempt events generated in 1 second (default heartbeat interval
for NM-RM heartbeat and AMRMClientAsync heartbeat to RM) in large cluster, then
we assume 10*1000 scheduler events could happens on rmDispatcher, then we can
estimate up to 10*(10*1000) events (include other events than SchedulerEvent)
could happens per second there. Based on this assumption, if we want to warn
ahead of 10 seconds before queue get full (assume peek operations get slow), so
may be 10 (seconds) * 10 (event types on rmScheduler) * (10*1000) (scale of
Nodes and Apps / interval) sounds like a reasonable value here?
In addition, I think we should fix tiny issue in below code (qSize % 1000 == 0)
doesn't make sense as qSize default to be 2^32 -1:
{code}
int qSize = eventQueue.size();
if (qSize !=0 qSize %1000 == 0) {
LOG.info(Size of event-queue is + qSize);
}
int remCapacity = eventQueue.remainingCapacity();
if (remCapacity 1000) {
LOG.warn(Very low remaining capacity in the event-queue:
+ remCapacity);
}
{code}
Dispatcher warn message is too late
---
Key: YARN-402
URL: https://issues.apache.org/jira/browse/YARN-402
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Lohit Vijayarenu
Priority: Minor
AsyncDispatcher throws out Warn when capacity remaining is less than 1000
{noformat}
if (remCapacity 1000) {
LOG.warn(Very low remaining capacity in the event-queue:
+ remCapacity);
}
{noformat}
What would be useful is to warn much before that, may be half full instead of
when queue is completely full. I see that eventQueue capacity is int value.
So, if one warn's queue has only 1000 capacity left, then service definitely
has serious problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-402) Dispatcher warn message is too late
[
https://issues.apache.org/jira/browse/YARN-402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395408#comment-14395408
]
Junping Du commented on YARN-402:
-
Forget that the queue can be constructed with other queue type other than
LinkedBlockingQueue. So may be it could be the smaller value within half of
queue size (if queue is not LinkedBlockingQueue by default) and 1000* 1000
(estimated as above).
Dispatcher warn message is too late
---
Key: YARN-402
URL: https://issues.apache.org/jira/browse/YARN-402
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Lohit Vijayarenu
Priority: Minor
AsyncDispatcher throws out Warn when capacity remaining is less than 1000
{noformat}
if (remCapacity 1000) {
LOG.warn(Very low remaining capacity in the event-queue:
+ remCapacity);
}
{noformat}
What would be useful is to warn much before that, may be half full instead of
when queue is completely full. I see that eventQueue capacity is int value.
So, if one warn's queue has only 1000 capacity left, then service definitely
has serious problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
