[
https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15555895#comment-15555895
]
Wangda Tan commented on YARN-2009:
----------------------------------
bq. I am not comfortable with fixing this in another patch. Our main use case
is the one where multiple users need to use the same queue with apps at the
same priority...
[~eepayne], I'm not trying to argue the importance of the use case: {{multiple
users need to use the same queue with apps at the same priority}} -- it's the
most important use case of intra-queue preemption. But to make we can easier
review the patch, can we move it to a separate JIRA and start to work on it
right after the JIRA is get committed (if we're generally fine with existing
JIRA). This patch is already very dense, review a patch with changed logics is
much harder than review a separate patch on top of this one.
The major problem of why this patch cannot solve the preemption between user is
-- it doesn't calculate "ideal allocation" and "preemptable resource" for user.
Existing logic only caps calculated ideal allocation by user limit, we need to
do more:
For example, when there're two users in a queue, u1 above UL, u2 below UL,
however u2 doesn't have more pending resource, in this case, we shouldn't
preempt resource from u1.
To solve the problem, we need add some logics to
{{calculateIdealAssignedResourcePerApp}} to calculate the actual ideal
allocation / preemptable for users, and we need update the while condition in
{{validateOutSameAppPriorityFromDemand}} as well. It doesn't sound like a very
simple patch (few lines change) to me.
The change proposal mentioned in your above comment is not correct:
bq. I think it should be something like the following, maybe: ...
This will cause application from the same user preempt according to FIFO order
(A1/A2 from the same user, A1 submitted first, and ask for more resource, A2
can preempt resource from A2), which is what we want to avoid.
And this doesn't sound correct to me as well:
bq. I think that instead, it should order the apps by how much they are
underserved. I think that it should be ordering the apps by
tmpApp.toBePreemptedByOther instead of priority...
It could preempt violate FIFO order, and in the scheduling cycle, preempted
resource may come back to the app which is just get preempted.
Please let me if you have any concerns to move the user limit preemption to a
separate JIRA.
Thanks,
> Priority support for preemption in ProportionalCapacityPreemptionPolicy
> -----------------------------------------------------------------------
>
> Key: YARN-2009
> URL: https://issues.apache.org/jira/browse/YARN-2009
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
> Reporter: Devaraj K
> Assignee: Sunil G
> Attachments: YARN-2009.0001.patch, YARN-2009.0002.patch,
> YARN-2009.0003.patch, YARN-2009.0004.patch, YARN-2009.0005.patch
>
>
> While preempting containers based on the queue ideal assignment, we may need
> to consider preempting the low priority application containers first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
