[
https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15748766#comment-15748766
]
Jason Lowe commented on YARN-5889:
----------------------------------
+1 for keeping the user limit behavior consistent with what it does today.
Resources will still be offered to apps in priority order (typically FIFO
unless overridden by the app's priority), and any user under the minimum limit
should still be able to receive that resource. As such I'm not sure I agree
with the proposal for active user limits. It appears that limit can go below
the configured minimum if a lot of users are all asking for the available
resources. That would change the behavior and affect app priorities beyond
what the user limit does today. For purposes of unblocking the user limit
preemption work, I'd like to preserve the current computation so adding the
preemption feature doesn't affect the behavior of the scheduler even when the
feature isn't used.
Speaking of preemption, we will need to be careful on how the two limits are
applied to prevent oscillations. If resources are available then the active
user limit should be applied as it is today. However if resources are not
available then the preemption limit should be applied not only to the user
being targeted for preemption but also to the user who is requesting the
resource to trigger the preemption. If we mix the two limits then preemption
can oscillate between two users because the active limit can be larger than the
preemption limit.
For calculating the preemption all-user-limit, I think the equation can be
simplified to absolute queue capacity * MULP. I personally do not want
preemptions to give users resources beyond the minimum. For example if the
MULP is configured to 25% and there are two users in the queue, A at 70% usage
and B at 30%, I'd rather not lose work by shooting A's containers to give B
resources beyond the configured minimum limit. Preemptions can be very costly
to an application, so I don't think this should be completely fair (that's the
job of FairScheduler). We should only preempt work to get users up to the
minimum limit and only if others are above the minimum limit. Thoughts on this?
> Improve user-limit calculation in capacity scheduler
> ----------------------------------------------------
>
> Key: YARN-5889
> URL: https://issues.apache.org/jira/browse/YARN-5889
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Reporter: Sunil G
> Assignee: Sunil G
> Attachments: YARN-5889.v0.patch, YARN-5889.v1.patch,
> YARN-5889.v2.patch
>
>
> Currently user-limit is computed during every heartbeat allocation cycle with
> a write lock. To improve performance, this tickets is focussing on moving
> user-limit calculation out of heartbeat allocation flow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
