[
https://issues.apache.org/jira/browse/YARN-8379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522977#comment-16522977
]
Zian Chen commented on YARN-8379:
---------------------------------
[~eepayne], for the suggestions you mentioned before which is make the
balancing was done all at once instead of add FifoSelector twice. Let me
explain this in two aspects,
1. in TempQueuePerPartition#offer. When we calculate ideal assignment, first
time we calculate accepted using this,
{code:java}
// accepted = min{avail,
// max - assigned,
// current + pending - assigned,
// # Make sure a queue will not get more than max of its
// # used/guaranteed, this is to make sure preemption won't
// # happen if all active queues are beyond their guaranteed
// # This is for leaf queue only.
// max(guaranteed, used) - assigned}
{code}
The second time, we calculate accepted without check max(guaranteed, used), as
far as I can see, this two steps should be done sequentially instead of in one
shot.
2. Another reason is we add an option to set configureable timeout for
preempt-to-balance selected containers (selected by fifo2) which can let user
set custom timeout for these preempt-to-balance containers and actually kill
them faster/slower based on user needs, which leads to control the balance
process to be faster or slower. But this timeout should only be affects
containers selected for balance, not for an underutilized queue to reach its
guaranteed resource. So we need to separate these two process.
All other comments should be handled by the latest patch already. Thanks!
> Add an option to allow Capacity Scheduler preemption to balance satisfied
> queues
> --------------------------------------------------------------------------------
>
> Key: YARN-8379
> URL: https://issues.apache.org/jira/browse/YARN-8379
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Assignee: Zian Chen
> Priority: Major
> Attachments: YARN-8379.001.patch, YARN-8379.002.patch,
> YARN-8379.003.patch, YARN-8379.004.patch, ericpayne.confs.tgz
>
>
> Existing capacity scheduler only supports preemption for an underutilized
> queue to reach its guaranteed resource. In addition to that, there’s an
> requirement to get better balance between queues when all of them reach
> guaranteed resource but with different fairness resource.
> An example is, 3 queues with capacity, queue_a = 30%, queue_b = 30%, queue_c
> = 40%. At time T. queue_a is using 30%, queue_b is using 70%. Existing
> scheduler preemption won't happen. But this is unfair to queue_a since
> queue_a has the same guaranteed resources.
> Before YARN-5864, capacity scheduler do additional preemption to balance
> queues. We changed the logic since it could preempt too many containers
> between queues when all queues are satisfied.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
