[jira] [Commented] (YARN-2069) Add cross-user preemption within CapacityScheduler's leaf-queue
[ https://issues.apache.org/jira/browse/YARN-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14053959#comment-14053959 ] Mayank Bansal commented on YARN-2069: - hi [~wangda], Thanks for the review. I updated the patch please take a look , Le tme answer your questions. bq. In ProportionalCapacityPreemptionPolicy, bq. 1) balanceUserLimitsinQueueForPreemption() bq. 1.1, I think there's a bug when multiple applications under a same user (say Jim) in a queue, and usage of Jim is over user-limit. Any of Jim's applications will be tried to be preempted (total-resource-used-by-Jim - user-limit). We should remember resourcesToClaimBackFromUser and initialRes for each user (not reset them when handling each application) And it's better to add test to make sure this behavior is correct. We need to maintian the reverse order of application submission which only can be done iterating through applications as we want to preempt applications which are last submitted. bq. 1.2, Some debug logging should be removed like Done bq. 1.3, This check should be unnecessary Done bq. 2) preemptFrom bq. I noticed this method will be called multiple times for a same application within a editSchedule() call. bq. The reservedContainers will be calculated multiple times. bq. An alternative way to do this is to cache This method will only be executed for all the applicatoins only once as we will be removing all reservations and for the apps the reservation is been removed that would be no-op bq.In LeafQueue, bq. 1) I think it's better to remember user limit, no need to compute it every time, add a method like getUserLimit() to leafQueue should be better. That valus is not static and changed every time based on cluster utilization and thats why i am calculating every time. bq, 1) Should we preempt containers equally from users when there're multiple users beyond user-limit in a queue? Its not good it should be based on who submitted last and over user limit, however its not fair but we want to preempt last jobs first bq. 2) Should we preempt containers equally from applications in a same user? (Heap-like data structure maybe helpful to solve 1/2) No as above mentioned bq. 3) Should user-limit preemption be configurable? I think if we just configure preemption , thats enough thoughts? Thanks, Mayank Add cross-user preemption within CapacityScheduler's leaf-queue --- Key: YARN-2069 URL: https://issues.apache.org/jira/browse/YARN-2069 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Vinod Kumar Vavilapalli Assignee: Mayank Bansal Attachments: YARN-2069-trunk-1.patch, YARN-2069-trunk-2.patch, YARN-2069-trunk-3.patch Preemption today only works across queues and moves around resources across queues per demand and usage. We should also have user-level preemption within a queue, to balance capacity across users in a predictable manner. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2069) Add cross-user preemption within CapacityScheduler's leaf-queue
[
https://issues.apache.org/jira/browse/YARN-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050947#comment-14050947
]
Hadoop QA commented on YARN-2069:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12653724/YARN-2069-trunk-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4183//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4183//console
This message is automatically generated.
Add cross-user preemption within CapacityScheduler's leaf-queue
---
Key: YARN-2069
URL: https://issues.apache.org/jira/browse/YARN-2069
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler
Reporter: Vinod Kumar Vavilapalli
Assignee: Mayank Bansal
Attachments: YARN-2069-trunk-1.patch, YARN-2069-trunk-2.patch,
YARN-2069-trunk-3.patch
Preemption today only works across queues and moves around resources across
queues per demand and usage. We should also have user-level preemption within
a queue, to balance capacity across users in a predictable manner.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2069) Add cross-user preemption within CapacityScheduler's leaf-queue
[
https://issues.apache.org/jira/browse/YARN-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041845#comment-14041845
]
Wangda Tan commented on YARN-2069:
--
Hi Mayank,
Thanks for you patch, I just looked at it, some comments on implementation,
*In ProportionalCapacityPreemptionPolicy,*
1) balanceUserLimitsinQueueForPreemption()
1.1, I think there's a bug when multiple applications under a same user (say
Jim) in a queue, and usage of Jim is over user-limit.
Any of Jim's applications will be tried to be preempted
(total-resource-used-by-Jim - user-limit).
We should remember resourcesToClaimBackFromUser and initialRes for each user
(not reset them when handling each application)
And it's better to add test to make sure this behavior is correct.
1.2, Some debug logging should be removed like
{code}
LOG.info(MAYANK queue NAME + qT.leafQueue.getQueueName());
{code}
1.3, This check should be unnecessary
{code}
if(userLimitforQueue == null){
LOG.info(MAYANK USER LIMIT IS NULL);
}
{code}
2) preemptFrom
I noticed this method will be called multiple times for a same application
within a editSchedule() call.
The reservedContainers will be calculated multiple times.
An alternative way to do this is to cache
{code}
ListRMContainer containers =
new ArrayListRMContainer(app.getLiveContainers());
{code}
At the begining of getContainersToPreempt(), clone them to a set for these
running containers. And remove container from the set if we processed it.
Another benefit to do is eliminate checking like,
{code}
SetRMContainer userContainers = userLimitContainers.get(app
.getApplicationAttemptId());
if (userContainers.contains(c)) {
continue;
}
{code}
And preemptFrom called by balanceUserLimitsinQueueForPreemption passed dumb
skippedAMContainerlist/skippedAMSize in. It's better to create a overloaded
preemptFrom to make code can be easier read.
*In LeafQueue,*
1) I think it's better to remember user limit, no need to compute it every
time, add a method like getUserLimit() to leafQueue should be better.
*In beyond, questions about some undefined requirements,*
1) Should we preempt containers equally from users when there're multiple users
beyond user-limit in a queue?
2) Should we preempt containers equally from applications in a same user?
(Heap-like data structure maybe helpful to solve 1/2)
3) Should user-limit preemption be configurable?
The last is as [~sunilg] suggested, your patch is based on YARN-2022, it's
better to separate your patch from YARN-2022 for easier review. You can point
which patch in YARN-2022 you based on.
Thanks,
Wangda
Add cross-user preemption within CapacityScheduler's leaf-queue
---
Key: YARN-2069
URL: https://issues.apache.org/jira/browse/YARN-2069
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler
Reporter: Vinod Kumar Vavilapalli
Assignee: Mayank Bansal
Attachments: YARN-2069-trunk-1.patch
Preemption today only works across queues and moves around resources across
queues per demand and usage. We should also have user-level preemption within
a queue, to balance capacity across users in a predictable manner.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2069) Add cross-user preemption within CapacityScheduler's leaf-queue
[
https://issues.apache.org/jira/browse/YARN-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14042800#comment-14042800
]
Hadoop QA commented on YARN-2069:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652297/YARN-2069-trunk-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4067//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4067//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4067//console
This message is automatically generated.
Add cross-user preemption within CapacityScheduler's leaf-queue
---
Key: YARN-2069
URL: https://issues.apache.org/jira/browse/YARN-2069
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler
Reporter: Vinod Kumar Vavilapalli
Assignee: Mayank Bansal
Attachments: YARN-2069-trunk-1.patch, YARN-2069-trunk-2.patch
Preemption today only works across queues and moves around resources across
queues per demand and usage. We should also have user-level preemption within
a queue, to balance capacity across users in a predictable manner.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2069) Add cross-user preemption within CapacityScheduler's leaf-queue
[ https://issues.apache.org/jira/browse/YARN-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041617#comment-14041617 ] Mayank Bansal commented on YARN-2069: - Taking it over. Thanks, Mayank Add cross-user preemption within CapacityScheduler's leaf-queue --- Key: YARN-2069 URL: https://issues.apache.org/jira/browse/YARN-2069 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Vinod Kumar Vavilapalli Assignee: Mayank Bansal Attachments: YARN-2069-trunk-1.patch Preemption today only works across queues and moves around resources across queues per demand and usage. We should also have user-level preemption within a queue, to balance capacity across users in a predictable manner. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2069) Add cross-user preemption within CapacityScheduler's leaf-queue
[
https://issues.apache.org/jira/browse/YARN-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041650#comment-14041650
]
Hadoop QA commented on YARN-2069:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652121/YARN-2069-trunk-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4059//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4059//console
This message is automatically generated.
Add cross-user preemption within CapacityScheduler's leaf-queue
---
Key: YARN-2069
URL: https://issues.apache.org/jira/browse/YARN-2069
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler
Reporter: Vinod Kumar Vavilapalli
Assignee: Mayank Bansal
Attachments: YARN-2069-trunk-1.patch
Preemption today only works across queues and moves around resources across
queues per demand and usage. We should also have user-level preemption within
a queue, to balance capacity across users in a predictable manner.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2069) Add cross-user preemption within CapacityScheduler's leaf-queue
[ https://issues.apache.org/jira/browse/YARN-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041690#comment-14041690 ] Sunil G commented on YARN-2069: --- Hi [~mayank_bansal] I guess this patch also has the code changes of YARN-2022. I think this can be separated. Add cross-user preemption within CapacityScheduler's leaf-queue --- Key: YARN-2069 URL: https://issues.apache.org/jira/browse/YARN-2069 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Vinod Kumar Vavilapalli Assignee: Mayank Bansal Attachments: YARN-2069-trunk-1.patch Preemption today only works across queues and moves around resources across queues per demand and usage. We should also have user-level preemption within a queue, to balance capacity across users in a predictable manner. -- This message was sent by Atlassian JIRA (v6.2#6252)
