[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888967#comment-15888967 ] Carlo Curino commented on YARN-5385: [~seanpo03] thanks for addressing some of my comments, we are getting closer, though I still have some doubts. Ok on 1-4, and I am fine postponing 5 to separate JIRA (I agree it will require some more work, and probably API discussions). Ok on 6 via switch. Regarding 7, I think in general we assume that multiple agents can operate on a single plan, and the "coordination" was happening in the plan (see writelocks there, and double-checking whether ReservationAllocation fits). I think we should follow the same style of coordination, thus finding a way to grab locks on the plan for an atomic set of changes. The fact that we have only one agent per plan, is not strictly required, mostly there because the invocations are almost static. I suggest to keep the coordination in the plan. Also even if we do what you propose, the lock should only kick-in if you have to make room, and not for normal submissions. [~subru] I would like your opinion on this: What do you think about having an explicit grabbing and release of the plan writelocks (from the agent side)? In a sense we are providing begin-transaction, commit type methods. [~seanpo03] can you do a QPS test for this? this locks worry me a bit. Ok to do more complex solutions like in suggested in 8 in YARN-6226. Good on 9. I like the default semantics for 10. Good on 11. > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-hard > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v005.patch, YARN-5385.v006.patch, > YARN-5385.v007.patch, YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887042#comment-15887042 ] Sean Po commented on YARN-5385: --- Two of the test failures are accounted for with a JIRA. YARN-6231 - hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler::testMoveWouldViolateMaxResourcesConstraints YARN-5548 - org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart::testFinishedAppRemovalAfterRMRestart The remaining failure hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption::testPreemptionSelectNonAMContainer appears to be a flaky test. A local run of the test was very inconsistent, with some fails and some successes across multiple tests. YARN-6249 was created to track this problem. > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-hard > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v005.patch, YARN-5385.v006.patch, > YARN-5385.v007.patch, YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[
https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885833#comment-15885833
]
Hadoop QA commented on YARN-5385:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 47m 33s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5385 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854881/YARN-5385.v007.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b9f1548b6e1b 3.13.0-108-generic #155-Ubuntu SMP Wed
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[
https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882038#comment-15882038
]
Hadoop QA commented on YARN-5385:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
15s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 30s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| | Null passed for non-null parameter of addYieldedReservations(List, Plan,
ReservationId, boolean) in
org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PriorityReservationAgent.updateReservation(ReservationId,
String, Plan, ReservationDefinition) Method invoked at
PriorityReservationAgent.java:of addYieldedReservations(List, Plan,
ReservationId, boolean) in
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881843#comment-15881843 ] Sean Po commented on YARN-5385: --- The two test failures are: > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-hard > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v005.patch, YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[
https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881587#comment-15881587
]
Hadoop QA commented on YARN-5385:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
14s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 43s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
15s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| |
org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PriorityReservationAgent.createReservation(ReservationId,
String, Plan, ReservationDefinition) does not release lock on all paths At
PriorityReservationAgent.java:not release lock on all paths At
PriorityReservationAgent.java:[line 80] |
| |
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[
https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706987#comment-15706987
]
Carlo Curino commented on YARN-5385:
[~seanpo03] thanks for working on this.
I like the general direction, few issues below:
# in {{ReservationSchedulerConfiguration}} you implemented
{{getPriorityReservationAgent(String queue)}} as a fix return of
{{DEFAULT_PRIORITY_RESERVATION_AGENT_NAME}}. Why? I see either a lookup in the
config flies with fallback on this, or not even having the get-by-queue method.
As is it seems a bit redundant.
# minor: you use in many comments "accomodate for", which is a bit unusual
english I think.
# how likely is that we need to "make room" for another reservation? If it is
rather likely we might want to lower the verbosity of the logs in
{{PriorityReservationAgent}}
# {{SimplePriorityReservationAgent}} to where it is used in
{{PriorityReservationAgent.addYieldedReservations}}. The cost is the same, but
a different sub-class might not return the reservation in priority order.
Alternatively, you should very prominently document this in the API.
# in {{PriorityReservationAgent.addYieldedReservations}} you simply log the
once-accepted, now-rejected reservations. Is this the best we can do? Can we
retain the list (with ReservationDefinition), and make it accessible from the
outside and/or try to come up with a call-back mechanism of sort? Maybe upon
the delete of a reservation, we could attempt to re-add some of the rejected
ones? etc..
# One concern is that if the current reservation does not fit even after
removing the lower prioirty reservations, you might change the set of accepted
reservations (e.g., because the order you submit the is no more the FIFO in
which they were accepted but a priority order, which might lead to less
reservations to fit). I should probably follow the suggestion above, and have a
switch in addYieldedReservations to retain the order of acceptance (which you
should also make sure is updated when the submission of a higher priority item
scramble this set). Try to define precisely what is the invariant enforced
(beside "low-priority reservation never prevent a higher priority reservation
to be accepted").
# I don't understand the concurrency model. How are you protecting from
concurrent/clashing invocations of the priority-enabled agents?
# Using something like tombstones or logically excluding lower-priority
reservations from the used resources might allow you to deal with concurrency
as well as some of the other issues above.
# ReservationComparator should like deal with null inputs, and comparing long
could be done with a simple difference instead of the method you have.
# what happens if you have reservation priority turned on, but the
{{ReservationDefinition}} does not carry its value? (e.g., in {{RMWebService},
but also in general}
# I haven't look at tests carefully, but I like the fact that you have plenty
scenarios. In general, I like to have methods that checks invariants: e.g.,
that no lower-priority reservation is accepted when a higher priority could fit
in the same space, or that no reservation is rejected if there is room to
accepted etc... This also allows you to have large randomized tests, and simply
check the invariants for those (I find those very useful to catch funky corner
cases).
> Add a PriorityAgent in ReservationSystem
> -
>
> Key: YARN-5385
> URL: https://issues.apache.org/jira/browse/YARN-5385
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, fairscheduler, resourcemanager
>Reporter: Sean Po
>Assignee: Sean Po
> Labels: oct16-hard
> Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch,
> YARN-5385.v004.patch, YARN-5385.v1.patch
>
>
> YARN-5211 proposes adding support for generalized priorities for reservations
> in the YARN ReservationSystem. This JIRA is a sub-task to track the addition
> of a priority agent to accomplish it. Please refer to the design doc in the
> parent JIRA for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563668#comment-15563668 ] Sean Po commented on YARN-5385: --- The final Javac warning about unchecked conversion in RMWebServices is unrelated to the changes in YARN-5385.v004.patch. > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[
https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563240#comment-15563240
]
Hadoop QA commented on YARN-5385:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 31s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 54s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 54s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn generated 1 new + 35 unchanged -
0 fixed = 36 total (was 35) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch
generated 0 new + 204 unchanged - 1 fixed = 204 total (was 205) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 35m 34s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m 16s
{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 57s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832513/YARN-5385.v004.patch |
| JIRA Issue | YARN-5385 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8b5bc92ad6a2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / cef61d5 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/13339/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/13339/testReport/ |
| modules |
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[
https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556800#comment-15556800
]
Hadoop QA commented on YARN-5385:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 36s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 19s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 19s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn generated 1 new + 35 unchanged -
0 fixed = 36 total (was 35) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch
generated 0 new + 204 unchanged - 1 fixed = 204 total (was 205) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 7s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 35m 25s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m 10s
{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 13s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| |
org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.SimplePriorityReservationAgent$ReservationComparator
implements Comparator but not Serializable At
SimplePriorityReservationAgent.java:Serializable At
SimplePriorityReservationAgent.java:[lines 107-125] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12832217/YARN-5385.v003.patch |
| JIRA Issue | YARN-5385 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux daaa1608bf3f
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[
https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15544077#comment-15544077
]
Hadoop QA commented on YARN-5385:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 56s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 31s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 21s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 21s {color}
| {color:red} hadoop-yarn-project_hadoop-yarn generated 1 new + 35 unchanged -
0 fixed = 36 total (was 35) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 41s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 1
new + 204 unchanged - 1 fixed = 205 total (was 205) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 3s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 35m 6s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 22s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 4s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| | Should
org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.SimplePriorityReservationAgent$ReservationComparator
be a _static_ inner class? At SimplePriorityReservationAgent.java:inner
class? At SimplePriorityReservationAgent.java:[lines 107-128] |
| Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMProxy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831427/YARN-5385.v002.patch |
| JIRA Issue | YARN-5385 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[
https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527554#comment-15527554
]
Hadoop QA commented on YARN-5385:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s {color}
| {color:red} YARN-5385 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830578/YARN-5385.v1.patch |
| JIRA Issue | YARN-5385 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/13231/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add a PriorityAgent in ReservationSystem
> -
>
> Key: YARN-5385
> URL: https://issues.apache.org/jira/browse/YARN-5385
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, fairscheduler, resourcemanager
>Reporter: Sean Po
>Assignee: Sean Po
> Attachments: YARN-5385.v1.patch
>
>
> YARN-5211 proposes adding support for generalized priorities for reservations
> in the YARN ReservationSystem. This JIRA is a sub-task to track the addition
> of a priority agent to accomplish it. Please refer to the design doc in the
> parent JIRA for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
