[jira] [Commented] (YARN-5516) Add REST API for periodicity
[ https://issues.apache.org/jira/browse/YARN-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220921#comment-16220921 ] Sean Po commented on YARN-5516: --- The following tests are failing but are tracked, known issues:. hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation tracked by YARN-6747 org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA tracked by YARN-7080 > Add REST API for periodicity > > > Key: YARN-5516 > URL: https://issues.apache.org/jira/browse/YARN-5516 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sangeetha Abdu Jyothi >Assignee: Sean Po > Attachments: YARN-5516.v001.patch, YARN-5516.v002.patch, > YARN-5516.v003.patch, YARN-5516.v004.patch, YARN-5516.v005.patch, > YARN-5516.v006.patch > > > YARN-5516 changing REST API of the reservation system to support periodicity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5516) Add REST API for periodicity
[ https://issues.apache.org/jira/browse/YARN-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5516: -- Attachment: YARN-5516.v006.patch Thanks [~subru] for the comments, the latest patch changes the dates in the InMemoryPlan tests to reference 2050 instead of 2017. > Add REST API for periodicity > > > Key: YARN-5516 > URL: https://issues.apache.org/jira/browse/YARN-5516 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sangeetha Abdu Jyothi >Assignee: Sean Po > Attachments: YARN-5516.v001.patch, YARN-5516.v002.patch, > YARN-5516.v003.patch, YARN-5516.v004.patch, YARN-5516.v005.patch, > YARN-5516.v006.patch > > > YARN-5516 changing REST API of the reservation system to support periodicity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5516) Add REST API for periodicity
[ https://issues.apache.org/jira/browse/YARN-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5516: -- Attachment: YARN-5516.v005.patch The following tests are failing but are also tracked. hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation tracked by YARN-6747 org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA tracked by YARN-7080 The following tests do not appear to be related to my changes. hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA The latest patch fixes the bug in the InMemoryPlan I referenced above, and also adds a test with regards to the bug. > Add REST API for periodicity > > > Key: YARN-5516 > URL: https://issues.apache.org/jira/browse/YARN-5516 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sangeetha Abdu Jyothi >Assignee: Sean Po > Attachments: YARN-5516.v001.patch, YARN-5516.v002.patch, > YARN-5516.v003.patch, YARN-5516.v004.patch, YARN-5516.v005.patch > > > YARN-5516 changing REST API of the reservation system to support periodicity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5516) Add REST API for periodicity
[ https://issues.apache.org/jira/browse/YARN-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218925#comment-16218925 ] Sean Po commented on YARN-5516: --- There is a bug in the code in line 639 of InMemoryPlan, where the duration can potentially be negative. I have added a fix along with a test to catch this error. I will update a patch when the Yetus run completes for the previous patch in case there is any action required as a result of a failed Yetus run. > Add REST API for periodicity > > > Key: YARN-5516 > URL: https://issues.apache.org/jira/browse/YARN-5516 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sangeetha Abdu Jyothi >Assignee: Sean Po > Attachments: YARN-5516.v001.patch, YARN-5516.v002.patch, > YARN-5516.v003.patch, YARN-5516.v004.patch > > > YARN-5516 changing REST API of the reservation system to support periodicity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5516) Add REST API for periodicity
[ https://issues.apache.org/jira/browse/YARN-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5516: -- Attachment: YARN-5516.v004.patch TestIncreaseAllocationExpirer tests fail intermittently and is tracked by YARN-7378. TestOpportunisticContainerAllocatorAMService also fails intermittently and is tracked by YARN-6841. The checkstyle issues above will be fixed in YARN-5516.v004.patch. > Add REST API for periodicity > > > Key: YARN-5516 > URL: https://issues.apache.org/jira/browse/YARN-5516 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sangeetha Abdu Jyothi >Assignee: Sean Po > Attachments: YARN-5516.v001.patch, YARN-5516.v002.patch, > YARN-5516.v003.patch, YARN-5516.v004.patch > > > YARN-5516 changing REST API of the reservation system to support periodicity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5516) Add REST API for periodicity
[ https://issues.apache.org/jira/browse/YARN-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5516: -- Attachment: YARN-5516.v003.patch Thanks [~subru] for the review. I added a change for all your comments. While adding the test for the RmWebServices, I found that there was a bug in the getReservations API of the InMemoryPlan. That bug has been fixed, and test cases were also created to catch future bugs in this area. > Add REST API for periodicity > > > Key: YARN-5516 > URL: https://issues.apache.org/jira/browse/YARN-5516 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sangeetha Abdu Jyothi >Assignee: Sean Po > Attachments: YARN-5516.v001.patch, YARN-5516.v002.patch, > YARN-5516.v003.patch > > > YARN-5516 changing REST API of the reservation system to support periodicity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7311) TestRMWebServicesReservation doesn't really test fair scheduler
[
https://issues.apache.org/jira/browse/YARN-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208154#comment-16208154
]
Sean Po commented on YARN-7311:
---
Thanks [~yufeigu] for working on this patch. The changes in the tests look good
to me, +1.
> TestRMWebServicesReservation doesn't really test fair scheduler
> ---
>
> Key: YARN-7311
> URL: https://issues.apache.org/jira/browse/YARN-7311
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, reservation system
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-7311.001.patch, YARN-7311.002.patch,
> YARN-7311.WIP.patch
>
>
> YARN-4248 introduced the REST API for submit/update/delete Reservations.
> Class {{TestRMWebServicesReservation}} intends to test both FS and CS. The
> test cases designed for fair scheduler actually test capacity scheduler. The
> following code in method {{configureServlets}} shows it sets the scheduler to
> CS even test cases are for fair scheduler.
> {code}
> conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
> ResourceScheduler.class);
> CapacitySchedulerConfiguration csconf =
> new CapacitySchedulerConfiguration(conf);
> String[] queues = { "default", "dedicated" };
> csconf.setQueues("root", queues);
> csconf.setCapacity("root.default", 50.0f);
> csconf.setCapacity("root.dedicated", 50.0f);
> csconf.setReservable("root.dedicated", true);
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5516) Add REST API for periodicity
[ https://issues.apache.org/jira/browse/YARN-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16205492#comment-16205492 ] Sean Po edited comment on YARN-5516 at 10/16/17 7:01 AM: - Verified that the tests that failed in the last run were unrelated to the patch. Added test to ensure the recurrence expression makes it all the way to the plan from REST API. was (Author: seanpo03): Verified that the tests that failed in the last run were unrelated to the patch. Added test to ensure the recurrence expression makes it all the way to the plan. > Add REST API for periodicity > > > Key: YARN-5516 > URL: https://issues.apache.org/jira/browse/YARN-5516 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sangeetha Abdu Jyothi >Assignee: Sean Po > Attachments: YARN-5516.v001.patch, YARN-5516.v002.patch > > > YARN-5516 changing REST API of the reservation system to support periodicity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5516) Add REST API for periodicity
[ https://issues.apache.org/jira/browse/YARN-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5516: -- Attachment: YARN-5516.v002.patch Verified that the tests that failed in the last run were unrelated to the patch. Added test to ensure the recurrence expression makes it all the way to the plan. > Add REST API for periodicity > > > Key: YARN-5516 > URL: https://issues.apache.org/jira/browse/YARN-5516 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sangeetha Abdu Jyothi >Assignee: Sean Po > Attachments: YARN-5516.v001.patch, YARN-5516.v002.patch > > > YARN-5516 changing REST API of the reservation system to support periodicity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5516) Add REST API for periodicity
[ https://issues.apache.org/jira/browse/YARN-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5516: -- Attachment: YARN-5516.v001.patch Added first version of the patch. > Add REST API for periodicity > > > Key: YARN-5516 > URL: https://issues.apache.org/jira/browse/YARN-5516 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sangeetha Abdu Jyothi >Assignee: Sean Po > Attachments: YARN-5516.v001.patch > > > YARN-5516 changing REST API of the reservation system to support periodicity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5329) ReservationAgent enhancements required to support recurring reservations in the YARN ReservationSystem
[
https://issues.apache.org/jira/browse/YARN-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181385#comment-16181385
]
Sean Po commented on YARN-5329:
---
[~subru], [~curino] the test failure occurs because of the change in
PlanningAlgorithm where _mapAllocations.put_ is swapped with
_allocation.addInterval_. I have applied your patch, and reverted that part of
the code, and the test cases that failed in TestRMWebServicesResevation passed.
{code:title=PlanningAlgorithm.java#allocationsToPaddedMap|borderStyle=solid}
// If period > 0...
// ...
} else {
// Pad at the beginning
long earliestStart = findEarliestTime(allocation.toIntervalMap());
if (jobArrival < earliestStart) {
allocation.addInterval(new ReservationInterval(jobArrival,
earliestStart), zeroResource);
}
// Pad at the beginning
long latestEnd = findLatestTime(allocation.toIntervalMap());
if (latestEnd < jobDeadline) {
allocation.addInterval(new ReservationInterval(latestEnd, jobDeadline),
zeroResource);
}
}
{code}
> ReservationAgent enhancements required to support recurring reservations in
> the YARN ReservationSystem
> --
>
> Key: YARN-5329
> URL: https://issues.apache.org/jira/browse/YARN-5329
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Subru Krishnan
>Assignee: Carlo Curino
> Attachments: YARN-5329.v0.patch, YARN-5329.v1.patch,
> YARN-5329.v2.patch, YARN-5329.v3.patch
>
>
> YARN-5326 proposes adding native support for recurring reservations in the
> YARN ReservationSystem. This JIRA is a sub-task to track the changes required
> in ReservationAgent to accomplish it. Please refer to the design doc in the
> parent JIRA for details.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043202#comment-16043202 ] Sean Po commented on YARN-6528: --- There are three check style failures: The first two are unavoidable without making a dramatic API change to InMemoryPlan. The last one is about about setting ReservationQueueMetrics to be a final class. This cannot be avoided because final classes cannot be mocked, and the following tests will fail: # hadoop.yarn.server.resourcemanager.reservation.planning.TestSimpleCapacityReplanner # hadoop.yarn.server.resourcemanager.reservation.TestInMemoryPlan # hadoop.yarn.server.resourcemanager.reservation.planning.TestGreedyReservationAgent # hadoop.yarn.server.resourcemanager.TestRMRestart # hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy # hadoop.yarn.server.resourcemanager.reservation.planning.TestAlignedPlanner # hadoop.yarn.server.resourcemanager.reservation.TestNoOverCommitPolicy > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch, > YARN-6528.v003.patch, YARN-6528.v004.patch, YARN-6528.v005.patch, > YARN-6528.v006.patch, YARN-6528.v007.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-6528: -- Attachment: YARN-6528.v007.patch YARN-6528.v007.patch addresses the checkstyle issues that could be fixed. There are a couple that had to be left unfixed purposefully. This new patch also prevents double counting when an agent that internally invokes another agent is used. > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch, > YARN-6528.v003.patch, YARN-6528.v004.patch, YARN-6528.v005.patch, > YARN-6528.v006.patch, YARN-6528.v007.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028717#comment-16028717 ] Sean Po commented on YARN-6528: --- The test failures are currently known flaky tests tracked below: YARN-5803 (testRMAppAttemptFailuresValidityInterval) YARN-5548 (testRMAppAttemptFailuresValidityInterval) The next patch will address the checkstyle issues. > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch, > YARN-6528.v003.patch, YARN-6528.v004.patch, YARN-6528.v005.patch, > YARN-6528.v006.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-6528: -- Attachment: YARN-6528.v006.patch YARN-6528.v006.patch fixes the remaining test failures. > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch, > YARN-6528.v003.patch, YARN-6528.v004.patch, YARN-6528.v005.patch, > YARN-6528.v006.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-6528: -- Attachment: YARN-6528.v005.patch YARN-6528.v005.patch fixes the findbugs error and the test failures. This patch has been tested on a single node cluster. > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch, > YARN-6528.v003.patch, YARN-6528.v004.patch, YARN-6528.v005.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-6528: -- Attachment: YARN-6528.v004.patch Thanks for the comments, [~curino]. I believe the YARN-6528.v004.patch is more in line with what you are looking for. > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch, > YARN-6528.v003.patch, YARN-6528.v004.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-6528: -- Attachment: YARN-6528.v003.patch YARN-6528.v003.patch fixes the issues that were brought up in the previous comment. > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch, > YARN-6528.v003.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008682#comment-16008682 ] Sean Po commented on YARN-6528: --- Thanks [~curino] for the comments. I'll fix 1 and 2 in the next iteration. 3. I think both can exist. Monitoring a single metric for abrupt changes to determine the system health is useful for automated alerts and watchdogs. Having metrics be posted on a per-plan basis is useful for users of the particular plan, as well as for administrators of the plan to determine if the system is in bad shape. What are your thoughts on this topic? 4. The rationale for keeping both the agent and plan metrics is to make it easier to determine the component responsible for a possible agent operation performance degradation. In addition, future agents may contain heavier placement logic, and it will be useful to have benchmark metrics to compare the agent metrics to. > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-6528: -- Attachment: YARN-6528.v002.patch YARN-6528.v002.patch fixes the findbugs and checkstyle issues that were present in the previous patch. The reason why these patches do not include unit tests is because no new logic is added with the addition of this patch. Instead, components are just being integrated with a set of metrics. > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001610#comment-16001610 ] Sean Po edited comment on YARN-6528 at 5/8/17 9:48 PM: --- YARN-6528.v002.patch fixes the findbugs and checkstyle issues that were present in the previous patch. The reason why these patches do not include unit tests is because no new logic is added with the addition of this patch. Instead, components are just being integrated with a set of metrics. To test this patch, a single node cluster was started, and reservations were submitted. Each of the added metrics were verified manually through the /jmx page. was (Author: seanpo03): YARN-6528.v002.patch fixes the findbugs and checkstyle issues that were present in the previous patch. The reason why these patches do not include unit tests is because no new logic is added with the addition of this patch. Instead, components are just being integrated with a set of metrics. > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-6528.v001.patch, YARN-6528.v002.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001302#comment-16001302 ] Sean Po edited comment on YARN-6528 at 5/8/17 6:58 PM: --- YARN-6528.v001.patch adds metric details to the ReservationSystem, and abstracts common functionality between AlignedPlannerWithGreedy and GreedyReservationAgent. Other implementations of ReservationAgent are not included in the patch because the other implementations are used within the AlignedPlannerWithGreedy and GreedyReservationAgent. This patch has been tested in a single node cluster. was (Author: seanpo03): YARN-6528.v001.patch adds metric details to the ReservationSystem, and abstracts common functionality between AlignedPlannerWithGreedy and GreedyReservationAgent. Other implementations of ReservationAgent are not included in the patch because the other implementations are used within the AlignedPlannerWithGreedy and GreedyReservationAgent. > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-6528.v001.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-6528: -- Attachment: YARN-6528.v001.patch YARN-6528.v001.patch adds metric details to the ReservationSystem, and abstracts common functionality between AlignedPlannerWithGreedy and GreedyReservationAgent. Other implementations of ReservationAgent are not included in the patch because the other implementations are used within the AlignedPlannerWithGreedy and GreedyReservationAgent. > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-6528.v001.patch > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6530) Add JMX metrics for Priority Reservation Agent
[ https://issues.apache.org/jira/browse/YARN-6530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po reassigned YARN-6530: - Assignee: Sean Po > Add JMX metrics for Priority Reservation Agent > -- > > Key: YARN-6530 > URL: https://issues.apache.org/jira/browse/YARN-6530 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed to gauge the performance of the priority reservation agent. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6529) Add JMX metrics for Priority Reservation Agent
[ https://issues.apache.org/jira/browse/YARN-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po resolved YARN-6529. --- Resolution: Duplicate Duplicate of YARN-6530. > Add JMX metrics for Priority Reservation Agent > -- > > Key: YARN-6529 > URL: https://issues.apache.org/jira/browse/YARN-6529 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed to gauge the performance of the priority reservation agent. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6530) Add JMX metrics for Priority Reservation Agent
Sean Po created YARN-6530: - Summary: Add JMX metrics for Priority Reservation Agent Key: YARN-6530 URL: https://issues.apache.org/jira/browse/YARN-6530 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sean Po YARN-5211 proposes adding support for generalized priorities for reservations in the YARN ReservationSystem. This JIRA is a sub-task to track the changes needed to gauge the performance of the priority reservation agent. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6529) Add JMX metrics for Priority Reservation Agent
[ https://issues.apache.org/jira/browse/YARN-6529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-6529: -- Description: YARN-5211 proposes adding support for generalized priorities for reservations in the YARN ReservationSystem. This JIRA is a sub-task to track the changes needed to gauge the performance of the priority reservation agent. > Add JMX metrics for Priority Reservation Agent > -- > > Key: YARN-6529 > URL: https://issues.apache.org/jira/browse/YARN-6529 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed to gauge the performance of the priority reservation agent. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6529) Add JMX metrics for Priority Reservation Agent
Sean Po created YARN-6529: - Summary: Add JMX metrics for Priority Reservation Agent Key: YARN-6529 URL: https://issues.apache.org/jira/browse/YARN-6529 Project: Hadoop YARN Issue Type: Task Reporter: Sean Po Assignee: Sean Po -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
[ https://issues.apache.org/jira/browse/YARN-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-6528: -- Description: YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle time explicitly, i.e. users can now "reserve" capacity ahead of time which is predictably allocated to them. In order to understand in finer detail the performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan Follower, Agent Placement and Plan Operations components of Rayon. > Add JMX metrics for Plan Follower and Agent Placement and Plan Operations > - > > Key: YARN-6528 > URL: https://issues.apache.org/jira/browse/YARN-6528 > Project: Hadoop YARN > Issue Type: Task >Reporter: Sean Po >Assignee: Sean Po > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. In order to understand in finer detail the > performance of Rayon, YARN-6528 proposes to include JMX metrics in the Plan > Follower, Agent Placement and Plan Operations components of Rayon. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6528) Add JMX metrics for Plan Follower and Agent Placement and Plan Operations
Sean Po created YARN-6528: - Summary: Add JMX metrics for Plan Follower and Agent Placement and Plan Operations Key: YARN-6528 URL: https://issues.apache.org/jira/browse/YARN-6528 Project: Hadoop YARN Issue Type: Task Reporter: Sean Po Assignee: Sean Po -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5331) Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968207#comment-15968207 ] Sean Po edited comment on YARN-5331 at 4/14/17 6:08 PM: Thanks for the patch [~ajsangeetha], it looks good to me. +1 YARN-5816 tracks TestDelegationTokenRenewer.testCancelWithMultipleAppSubmissions failure YARN-5548 tracks TestRMRestart.testFinishedAppRemovalAfterRMRestart failure was (Author: seanpo03): Thanks for the patch [~ajsangeetha], it looks good to me. +1 > Extend RLESparseResourceAllocation with period for supporting recurring > reservations in YARN ReservationSystem > -- > > Key: YARN-5331 > URL: https://issues.apache.org/jira/browse/YARN-5331 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Sangeetha Abdu Jyothi > Labels: oct16-medium > Attachments: YARN-5331.001.patch, YARN-5331.002.patch, > YARN-5331.003.patch, YARN-5331.004.patch, YARN-5331.005.patch, > YARN-5331.006.patch, YARN-5331.007.patch, YARN-5331.008.patch > > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to add a > PeriodicRLESparseResourceAllocation. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5331) Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968207#comment-15968207 ] Sean Po commented on YARN-5331: --- Thanks for the patch [~ajsangeetha], it looks good to me. +1 > Extend RLESparseResourceAllocation with period for supporting recurring > reservations in YARN ReservationSystem > -- > > Key: YARN-5331 > URL: https://issues.apache.org/jira/browse/YARN-5331 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Sangeetha Abdu Jyothi > Labels: oct16-medium > Attachments: YARN-5331.001.patch, YARN-5331.002.patch, > YARN-5331.003.patch, YARN-5331.004.patch, YARN-5331.005.patch, > YARN-5331.006.patch, YARN-5331.007.patch, YARN-5331.008.patch > > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to add a > PeriodicRLESparseResourceAllocation. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5331) Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939094#comment-15939094 ] Sean Po commented on YARN-5331: --- Thanks [~ajsangeetha] for the update, a few more comments: * It may be more appropriate to log instead of printing to stdout using System.out.println. (PeriodicRLESparseResourceAllocation.java+84,99) * NIT: PeriodicRLESparseResourceAllocation::removeInterval does not have the same checks for time interval that PeriodicRLESparseResourceAllocation::setCapacityInInterval does. * We have checks in PeriodicRLESparseResourceAllocation::setCapacityInInterval and PeriodicRLESparseResourceAllocation::removeInterval that will prevent the capacity from being added or removed. Can we verify these checks by adding test cases that would fail to be added or removed? * Would it be better to have PeriodicRLESparseResourceAllocation::setCapacityInInterval or PeriodicRLESparseResourceAllocation::removeInterval return bool, to allow the caller to determine whether or not the method call succeeded? * NIT: Please add a test case in PeriodicRLESparseResourceAllocation::testPeriodicCapacity, PeriodicRLESparseResourceAllocation::testRemoveInterval that does not land on a timeStep. * Do we need to handle the possibility of negative periodic capacity when PeriodicRLESparseResourceAllocation::removeInterval is used? > Extend RLESparseResourceAllocation with period for supporting recurring > reservations in YARN ReservationSystem > -- > > Key: YARN-5331 > URL: https://issues.apache.org/jira/browse/YARN-5331 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Sangeetha Abdu Jyothi > Labels: oct16-medium > Attachments: YARN-5331.001.patch, YARN-5331.002.patch, > YARN-5331.003.patch, YARN-5331.004.patch, YARN-5331.005.patch > > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to add a > PeriodicRLESparseResourceAllocation. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5703) ReservationAgents are not correctly configured
[ https://issues.apache.org/jira/browse/YARN-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898630#comment-15898630 ] Sean Po commented on YARN-5703: --- I apologize for not responding, [~maniraj...@gmail.com] and [~Naganarasimha]. I haven't been checking my email for the last few weeks for personal reasons. Thank you both for contributing, and checking this in! > ReservationAgents are not correctly configured > -- > > Key: YARN-5703 > URL: https://issues.apache.org/jira/browse/YARN-5703 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Sean Po >Assignee: Manikandan R > Attachments: YARN-5703.001.patch, YARN-5703.002.patch, > YARN-5703.003.patch, YARN-5703.004.patch, YARN-5703.005.patch, > YARN-5703.006.patch, YARN-5703.007.patch, YARN-5703.008.patch > > > In AbstractReservationSystem, the method that instantiates a ReservationAgent > does not properly initialize it with the appropriate configuration because it > expects the ReservationAgent to implement Configurable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5881) Enable configuration of queue capacity in terms of absolute resources
[ https://issues.apache.org/jira/browse/YARN-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po reassigned YARN-5881: - Assignee: Wangda Tan (was: Sean Po) > Enable configuration of queue capacity in terms of absolute resources > - > > Key: YARN-5881 > URL: https://issues.apache.org/jira/browse/YARN-5881 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sean Po >Assignee: Wangda Tan > Attachments: > YARN-5881.Support.Absolute.Min.Max.Resource.In.Capacity.Scheduler.design-doc.v1.pdf > > > Currently, Yarn RM supports the configuration of queue capacity in terms of a > proportion to cluster capacity. In the context of Yarn being used as a public > cloud service, it makes more sense if queues can be configured absolutely. > This will allow administrators to set usage limits more concretely and > simplify customer expectations for cluster allocation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5881) Enable configuration of queue capacity in terms of absolute resources
[ https://issues.apache.org/jira/browse/YARN-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898625#comment-15898625 ] Sean Po commented on YARN-5881: --- Thanks Wangda, please take over the ticket. I have assigned it to you. > Enable configuration of queue capacity in terms of absolute resources > - > > Key: YARN-5881 > URL: https://issues.apache.org/jira/browse/YARN-5881 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sean Po >Assignee: Sean Po > Attachments: > YARN-5881.Support.Absolute.Min.Max.Resource.In.Capacity.Scheduler.design-doc.v1.pdf > > > Currently, Yarn RM supports the configuration of queue capacity in terms of a > proportion to cluster capacity. In the context of Yarn being used as a public > cloud service, it makes more sense if queues can be configured absolutely. > This will allow administrators to set usage limits more concretely and > simplify customer expectations for cluster allocation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887042#comment-15887042 ] Sean Po commented on YARN-5385: --- Two of the test failures are accounted for with a JIRA. YARN-6231 - hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler::testMoveWouldViolateMaxResourcesConstraints YARN-5548 - org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart::testFinishedAppRemovalAfterRMRestart The remaining failure hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption::testPreemptionSelectNonAMContainer appears to be a flaky test. A local run of the test was very inconsistent, with some fails and some successes across multiple tests. YARN-6249 was created to track this problem. > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-hard > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v005.patch, YARN-5385.v006.patch, > YARN-5385.v007.patch, YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6249) TestFairSchedulerPreemption is inconsistently failing on trunk
[ https://issues.apache.org/jira/browse/YARN-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-6249: -- Affects Version/s: 2.9.0 > TestFairSchedulerPreemption is inconsistently failing on trunk > -- > > Key: YARN-6249 > URL: https://issues.apache.org/jira/browse/YARN-6249 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler, resourcemanager >Affects Versions: 2.9.0 >Reporter: Sean Po > > Tests in TestFairSchedulerPreemption.java will inconsistently fail on trunk. > An example stack trace: > Tests run: 24, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 24.879 sec > <<< FAILURE! - in > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption > testPreemptionSelectNonAMContainer[MinSharePreemptionWithDRF](org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption) > Time elapsed: 10.475 sec <<< FAILURE! > java.lang.AssertionError: Incorrect number of containers on the greedy app > expected:<4> but was:<8> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyPreemption(TestFairSchedulerPreemption.java:288) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.testPreemptionSelectNonAMContainer(TestFairSchedulerPreemption.java:363) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6249) TestFairSchedulerPreemption is inconsistently failing on trunk
Sean Po created YARN-6249: - Summary: TestFairSchedulerPreemption is inconsistently failing on trunk Key: YARN-6249 URL: https://issues.apache.org/jira/browse/YARN-6249 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler, resourcemanager Reporter: Sean Po Tests in TestFairSchedulerPreemption.java will inconsistently fail on trunk. An example stack trace: Tests run: 24, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 24.879 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption testPreemptionSelectNonAMContainer[MinSharePreemptionWithDRF](org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption) Time elapsed: 10.475 sec <<< FAILURE! java.lang.AssertionError: Incorrect number of containers on the greedy app expected:<4> but was:<8> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyPreemption(TestFairSchedulerPreemption.java:288) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.testPreemptionSelectNonAMContainer(TestFairSchedulerPreemption.java:363) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5385: -- Attachment: YARN-5385.v007.patch YARN-5385.v007.patch fixes a configuration bug found while running a single node cluster test. The patch also attempts to fix the find bugs error from the latest QA run. > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-hard > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v005.patch, YARN-5385.v006.patch, > YARN-5385.v007.patch, YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5385: -- Attachment: YARN-5385.v006.patch YARN-5385.v006.patch attempts to fix the findbugs errors by wrapping the methods being locked, with a try finally block. > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-hard > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v005.patch, YARN-5385.v006.patch, > YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881843#comment-15881843 ] Sean Po edited comment on YARN-5385 at 2/24/17 3:03 AM: The two test failures are known test failures tracked in JIRA: YARN-5548 - org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart testFinishedAppRemovalAfterRMRestart YARN-5816 - org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer testCancelWithMultipleAppSubmissions was (Author: seanpo03): The two test failures are: > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-hard > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v005.patch, YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881843#comment-15881843 ] Sean Po commented on YARN-5385: --- The two test failures are: > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-hard > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v005.patch, YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5385: -- Attachment: YARN-5385.v005.patch Thanks [~curino] for the review! I have addressed most of the comments you have given me. My comments below: 1. The CapacitySchedulerConfiguration implements the get by queue method. 2. I have changed the words 'accommodate for' in code and comments to 'make room for'. 3. In our clusters, we currently do not see many priority inversions. For our use, it seems that this log won't be too verbose, and having the logs will be more beneficial than harmful. 4. The makeRoomForReservation method is abstract for PriorityReservationAgent because subclasses will want to determine 2 things. 1, the order for deletion, and 2, the order for re-addition. I have documented this in the java doc to make it more clear that makeRoomForReservation is meant to return an ordered list of reservation allocations. 5. This problem will require a lot of work, and it may be more appropriate for this to be completed in a different task. I have created YARN-6227 to track this feature. 7. I have added locks around the implementation of the ReservationAgent API for the PriorityReservationAgent. I understand this solution may not be the best in the event that we get a lot of multi-threaded access in a short period of time, but a more appropriate solution will be complicated, and it may be better if it is done in a separate task. YARN-6226 was created to track this improvement. 10. In RMWebService, if the priority is not given, it will be set to an undefined value. PriorityReservationAgent converts undefined priority to the lowest possible priority. 11. I do have tests that check invariants. > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-hard > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v005.patch, YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6227) Information about reservations that are yielded by higher priority reservations need to be surfaced to the client
Sean Po created YARN-6227: - Summary: Information about reservations that are yielded by higher priority reservations need to be surfaced to the client Key: YARN-6227 URL: https://issues.apache.org/jira/browse/YARN-6227 Project: Hadoop YARN Issue Type: Task Reporter: Sean Po YARN-5385 introduces a PriorityReservationAgent that can be used to yield lower priority reservations so that higher priority reservations may fit if it initially could not be created in the plan. In the event that a low priority reservation is yielded, the client does not have a way to know prior to using the reservation that it has been removed. This JIRA aims to surface such information to the client, so that it may be able to act appropriately to its lost reservation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6226) Improve concurrency model in PriorityReservationAgent
Sean Po created YARN-6226: - Summary: Improve concurrency model in PriorityReservationAgent Key: YARN-6226 URL: https://issues.apache.org/jira/browse/YARN-6226 Project: Hadoop YARN Issue Type: Task Components: resourcemanager Reporter: Sean Po Assignee: Sean Po YARN-5385 introduces a PriorityReservationAgent that can be used to yield lower priority reservations so that higher priority reservations may fit if it initially could not be created in the plan. In YARN-5385, a write lock was used in the createReservation, updateReservation and deleteReservation methods to ensure that multi-threaded access will not be a problem for the new agent. This solution may be problematic if many multi-threaded reservation operations are made in a short period of time due to the heaviness of the createReservation and updateReservation methods. This JIRA aims to improve the concurrency model of the PriorityReservationAgent by replacing the write lock with a more suitable solution. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5703) ReservationAgents are not correctly configured
[ https://issues.apache.org/jira/browse/YARN-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840458#comment-15840458 ] Sean Po commented on YARN-5703: --- Hi [~maniraj...@gmail.com], thanks for your interest in this. Please feel free to work on this JIRA. > ReservationAgents are not correctly configured > -- > > Key: YARN-5703 > URL: https://issues.apache.org/jira/browse/YARN-5703 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Sean Po >Assignee: Sean Po > > In AbstractReservationSystem, the method that instantiates a ReservationAgent > does not properly initialize it with the appropriate configuration because it > expects the ReservationAgent to implement Configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5839) ClusterApplication API does not include a ReservationID
[ https://issues.apache.org/jira/browse/YARN-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5839: -- Attachment: YARN-5839.v003.patch YARN-5839.v003.patch fixes unit tests, Javadoc, and checkstyle. There is one checkstyle that remains. It cannot be avoided without unrelated refactoring. > ClusterApplication API does not include a ReservationID > --- > > Key: YARN-5839 > URL: https://issues.apache.org/jira/browse/YARN-5839 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5839.v001.patch, YARN-5839.v002.patch, > YARN-5839.v003.patch > > > Currently, the ClusterApplication, and ClusterApplications API does not allow > users to find the reservation queue that an application is running in. > YARN-5839 proposes to add ReservationId to the ClusterApplication and > ClusterApplications API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5839) ClusterApplication API does not include a ReservationID
[ https://issues.apache.org/jira/browse/YARN-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5839: -- Attachment: YARN-5839.v002.patch Re-create the patch from latest trunk. > ClusterApplication API does not include a ReservationID > --- > > Key: YARN-5839 > URL: https://issues.apache.org/jira/browse/YARN-5839 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5839.v001.patch, YARN-5839.v002.patch > > > Currently, the ClusterApplication, and ClusterApplications API does not allow > users to find the reservation queue that an application is running in. > YARN-5839 proposes to add ReservationId to the ClusterApplication and > ClusterApplications API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5435) [Regression] QueueCapacities not being updated for dynamic ReservationQueue
[ https://issues.apache.org/jira/browse/YARN-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5435: -- Attachment: YARN-5435.v005.patch YARN-5435.v005.patch modifies the code so that all tests pass, and only max QueueCapacities are inherited from the parent queue. > [Regression] QueueCapacities not being updated for dynamic ReservationQueue > --- > > Key: YARN-5435 > URL: https://issues.apache.org/jira/browse/YARN-5435 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 2.8.0 >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-easy, regression > Attachments: YARN-5435.v003.patch, YARN-5435.v004.patch, > YARN-5435.v005.patch, YARN-5435.v1.patch, YARN-5435.v2.patch > > > YARN-1707 added dynamic queues (ReservationQueue) to CapacityScheduler. The > QueueCapacities data structure was added subsequently but is not being > updated correctly for ReservationQueue. This JIRA tracks the changes required > to update QueueCapacities of ReservationQueue correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5839) ClusterApplication API does not include a ReservationID
[ https://issues.apache.org/jira/browse/YARN-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5839: -- Description: Currently, the ClusterApplication, and ClusterApplications API does not allow users to find the reservation queue that an application is running in. YARN-5839 proposes to add ReservationId to the ClusterApplication and ClusterApplications API. (was: Currently, the ClusterApplication, and ClusterApplications API does not allow users to find the reservation queue that an application is running in. YARN-5435 proposes to add ReservationId to the ClusterApplication and ClusterApplications API.) > ClusterApplication API does not include a ReservationID > --- > > Key: YARN-5839 > URL: https://issues.apache.org/jira/browse/YARN-5839 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5839.v001.patch > > > Currently, the ClusterApplication, and ClusterApplications API does not allow > users to find the reservation queue that an application is running in. > YARN-5839 proposes to add ReservationId to the ClusterApplication and > ClusterApplications API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5839) ClusterApplication API does not include a ReservationID
[ https://issues.apache.org/jira/browse/YARN-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703855#comment-15703855 ] Sean Po edited comment on YARN-5839 at 11/29/16 1:53 AM: - YARN-5839.v001.patch includes the changes to the REST and RPC API to add a ReservationId to the ApplicationReport. was (Author: seanpo03): YARN-5389.v001.patch includes the changes to the REST and RPC API to add a ReservationId to the ApplicationReport. > ClusterApplication API does not include a ReservationID > --- > > Key: YARN-5839 > URL: https://issues.apache.org/jira/browse/YARN-5839 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5839.v001.patch > > > Currently, the ClusterApplication, and ClusterApplications API does not allow > users to find the reservation queue that an application is running in. > YARN-5435 proposes to add ReservationId to the ClusterApplication and > ClusterApplications API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5839) ClusterApplication API does not include a ReservationID
[ https://issues.apache.org/jira/browse/YARN-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5839: -- Attachment: YARN-5839.v001.patch > ClusterApplication API does not include a ReservationID > --- > > Key: YARN-5839 > URL: https://issues.apache.org/jira/browse/YARN-5839 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5839.v001.patch > > > Currently, the ClusterApplication, and ClusterApplications API does not allow > users to find the reservation queue that an application is running in. > YARN-5435 proposes to add ReservationId to the ClusterApplication and > ClusterApplications API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5881) Enable configuration of queue capacity in terms of absolute resources
Sean Po created YARN-5881: - Summary: Enable configuration of queue capacity in terms of absolute resources Key: YARN-5881 URL: https://issues.apache.org/jira/browse/YARN-5881 Project: Hadoop YARN Issue Type: Improvement Reporter: Sean Po Assignee: Sean Po Currently, Yarn RM supports the configuration of queue capacity in terms of a proportion to cluster capacity. In the context of Yarn being used as a public cloud service, it makes more sense if queues can be configured absolutely. This will allow administrators to set usage limits more concretely and simplify customer expectations for cluster allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5839) ClusterApplication API does not include a ReservationID
Sean Po created YARN-5839: - Summary: ClusterApplication API does not include a ReservationID Key: YARN-5839 URL: https://issues.apache.org/jira/browse/YARN-5839 Project: Hadoop YARN Issue Type: Improvement Reporter: Sean Po Assignee: Sean Po Currently, the ClusterApplication, and ClusterApplications API does not allow users to find the reservation queue that an application is running in. YARN-5435 proposes to add ReservationId to the ClusterApplication and ClusterApplications API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5435) [Regression] QueueCapacities not being updated for dynamic ReservationQueue
[ https://issues.apache.org/jira/browse/YARN-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5435: -- Attachment: YARN-5435.v004.patch Thanks [~curino]. I added the change you suggested, and updated the tests to pass in the lastest change. > [Regression] QueueCapacities not being updated for dynamic ReservationQueue > --- > > Key: YARN-5435 > URL: https://issues.apache.org/jira/browse/YARN-5435 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 2.8.0 >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-easy, regression > Attachments: YARN-5435.v003.patch, YARN-5435.v004.patch, > YARN-5435.v1.patch, YARN-5435.v2.patch > > > YARN-1707 added dynamic queues (ReservationQueue) to CapacityScheduler. The > QueueCapacities data structure was added subsequently but is not being > updated correctly for ReservationQueue. This JIRA tracks the changes required > to update QueueCapacities of ReservationQueue correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5435) [Regression] QueueCapacities not being updated for dynamic ReservationQueue
[ https://issues.apache.org/jira/browse/YARN-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5435: -- Attachment: YARN-5435.v003.patch Thanks [~curino] for the review. This latest patch uses the CSQueueUtils class to inherit properties from the parent queue. It also adds the ability to whitelist labels for inheritance so that we can select just NO_LABEL to inherit. > [Regression] QueueCapacities not being updated for dynamic ReservationQueue > --- > > Key: YARN-5435 > URL: https://issues.apache.org/jira/browse/YARN-5435 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 2.8.0 >Reporter: Sean Po >Assignee: Sean Po > Labels: oct16-easy, regression > Attachments: YARN-5435.v003.patch, YARN-5435.v1.patch, > YARN-5435.v2.patch > > > YARN-1707 added dynamic queues (ReservationQueue) to CapacityScheduler. The > QueueCapacities data structure was added subsequently but is not being > updated correctly for ReservationQueue. This JIRA tracks the changes required > to update QueueCapacities of ReservationQueue correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5600: -- Labels: oct16-medium (was: ) > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5705) [YARN-3368] Add support for Timeline V2 to new web UI
[ https://issues.apache.org/jira/browse/YARN-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5705: -- Labels: oct16-hard (was: ) > [YARN-3368] Add support for Timeline V2 to new web UI > - > > Key: YARN-5705 > URL: https://issues.apache.org/jira/browse/YARN-5705 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Akhil PB > Labels: oct16-hard > Attachments: YARN-5705.001.patch, YARN-5705.002.patch, > YARN-5705.003.patch, YARN-5705.004.patch, YARN-5705.005.patch, > YARN-5705.006.patch, YARN-5705.007.patch, YARN-5705.008.patch, > YARN-5705.009.patch, YARN-5705.010.patch, YARN-5705.011.patch, > YARN-5705.012.patch > > > Integrate timeline v2 to YARN-3368. This is a clone JIRA for YARN-4097 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5722) FairScheduler hides group resolution exceptions when assigning queue
[
https://issues.apache.org/jira/browse/YARN-5722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Po updated YARN-5722:
--
Labels: oct16-easy (was: )
> FairScheduler hides group resolution exceptions when assigning queue
> -
>
> Key: YARN-5722
> URL: https://issues.apache.org/jira/browse/YARN-5722
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 2.6.5, 3.0.0-alpha1
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
> Labels: oct16-easy
> Attachments: YARN-5722.1.patch
>
>
> When a group based placement rule is used and the user does not have any
> groups the reason for rejecting the application is hidden. An assignment will
> fail as follows:
> {code}
>
>
> {code}
> The error logged on the client side:
> {code}
> 09/30 15:59:27 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /user/test_user/.staging/job_1475223610304_6043
> 16/09/30 15:59:27 WARN security.UserGroupInformation:
> PriviledgedActionException as:test_user (auth:SIMPLE)
> cause:java.io.IOException: Failed to run job : Error assigning app to queue
> default
> java.io.IOException: Failed to run job : Error assigning app to queue default
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
>
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
> {code}
> The {{default}} queue name is passed in as part of the application submission
> and not really the queue that is tried.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5767) Fix the order that resources are cleaned up from the local Public/Private caches
[
https://issues.apache.org/jira/browse/YARN-5767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Po updated YARN-5767:
--
Labels: oct16-easy (was: )
> Fix the order that resources are cleaned up from the local Public/Private
> caches
>
>
> Key: YARN-5767
> URL: https://issues.apache.org/jira/browse/YARN-5767
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.0, 3.0.0-alpha1
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Labels: oct16-easy
> Attachments: YARN-5767-trunk-v1.patch, YARN-5767-trunk-v2.patch,
> YARN-5767-trunk-v3.patch, YARN-5767-trunk-v4.patch
>
>
> If you look at {{ResourceLocalizationService#handleCacheCleanup}}, you can
> see that public resources are added to the {{ResourceRetentionSet}} first
> followed by private resources:
> {code:java}
> private void handleCacheCleanup(LocalizationEvent event) {
> ResourceRetentionSet retain =
> new ResourceRetentionSet(delService, cacheTargetSize);
> retain.addResources(publicRsrc);
> if (LOG.isDebugEnabled()) {
> LOG.debug("Resource cleanup (public) " + retain);
> }
> for (LocalResourcesTracker t : privateRsrc.values()) {
> retain.addResources(t);
> if (LOG.isDebugEnabled()) {
> LOG.debug("Resource cleanup " + t.getUser() + ":" + retain);
> }
> }
> //TODO Check if appRsrcs should also be added to the retention set.
> }
> {code}
> Unfortunately, if we look at {{ResourceRetentionSet#addResources}} we see
> that this means public resources are deleted first until the target cache
> size is met:
> {code:java}
> public void addResources(LocalResourcesTracker newTracker) {
> for (LocalizedResource resource : newTracker) {
> currentSize += resource.getSize();
> if (resource.getRefCount() > 0) {
> // always retain resources in use
> continue;
> }
> retain.put(resource, newTracker);
> }
> for (Iterator> i =
> retain.entrySet().iterator();
>currentSize - delSize > targetSize && i.hasNext();) {
> Map.Entry rsrc = i.next();
> LocalizedResource resource = rsrc.getKey();
> LocalResourcesTracker tracker = rsrc.getValue();
> if (tracker.remove(resource, delService)) {
> delSize += resource.getSize();
> i.remove();
> }
> }
> }
> {code}
> The result of this is that resources in the private cache are only deleted in
> the cases where:
> # The cache size is larger than the target cache size and the public cache is
> empty.
> # The cache size is larger than the target cache size and everything in the
> public cache is being used by a running container.
> For clusters that primarily use the public cache (i.e. make use of the shared
> cache), this means that the most commonly used resources can be deleted
> before old resources in the private cache. Furthermore, the private cache can
> continue to grow over time causing more and more churn in the public cache.
> Additionally, the same problem exists within the private cache. Since
> resources are added to the retention set on a user by user basis, resources
> will get cleaned up one user at a time in the order that privateRsrc.values()
> returns the LocalResourcesTracker. So if user1 has 10MB in their cache and
> user2 has 100MB in their cache and the target size of the cache is 50MB,
> user1 could potentially have their entire cache removed before anything is
> deleted from the user2 cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5763) HttpListener takes upwards of 3 minutes to start on failover
Sean Po created YARN-5763: - Summary: HttpListener takes upwards of 3 minutes to start on failover Key: YARN-5763 URL: https://issues.apache.org/jira/browse/YARN-5763 Project: Hadoop YARN Issue Type: Bug Reporter: Sean Po Assignee: Sean Po When Yarn RM fails over to another instance, it takes multiple minutes before the new master Yarn RM can begin accepting requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563668#comment-15563668 ] Sean Po commented on YARN-5385: --- The final Javac warning about unchecked conversion in RMWebServices is unrelated to the changes in YARN-5385.v004.patch. > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5385: -- Attachment: YARN-5385.v004.patch YARN-5385.v004.patch implements Serializable for the ReservationComparator. > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v004.patch, YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5385: -- Attachment: YARN-5385.v003.patch YARN-5385.v003.patch cleans up the test for the SimplePriorityReservationAgent, and adds tests for different ReservationPriorityScopes. > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5385.v002.patch, YARN-5385.v003.patch, > YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5385: -- Attachment: YARN-5385.v002.patch YARN-5385.v002.patch adds a priority aware reservation agent. I have tested this code on a single node cluster. > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5385.v002.patch, YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5703) ReservationAgents are not correctly configured
Sean Po created YARN-5703: - Summary: ReservationAgents are not correctly configured Key: YARN-5703 URL: https://issues.apache.org/jira/browse/YARN-5703 Project: Hadoop YARN Issue Type: Bug Components: capacity scheduler, resourcemanager Affects Versions: 3.0.0-alpha1 Reporter: Sean Po Assignee: Sean Po In AbstractReservationSystem, the method that instantiates a ReservationAgent does not properly initialize it with the appropriate configuration because it expects the ReservationAgent to implement Configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537141#comment-15537141 ] Sean Po commented on YARN-5384: --- Test failure is known: https://issues.apache.org/jira/browse/YARN-5043?jql=text%20~%20%22testRMAppAttemptFailuresValidityInterval%22. I also ran locally, and saw that the test ran successfully. > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch, YARN-5384.v10.patch, > YARN-5384.v11.patch, YARN-5384.v2.patch, YARN-5384.v3.patch, > YARN-5384.v4.patch, YARN-5384.v5.patch, YARN-5384.v6.patch, > YARN-5384.v7.patch, YARN-5384.v8.patch, YARN-5384.v9.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5384: -- Attachment: YARN-5384.v11.patch YARN-5384.v11.patch removes the remaining checkstyle issue. > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch, YARN-5384.v10.patch, > YARN-5384.v11.patch, YARN-5384.v2.patch, YARN-5384.v3.patch, > YARN-5384.v4.patch, YARN-5384.v5.patch, YARN-5384.v6.patch, > YARN-5384.v7.patch, YARN-5384.v8.patch, YARN-5384.v9.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534925#comment-15534925 ] Sean Po edited comment on YARN-5384 at 9/30/16 3:47 AM: YARN-5384.v10.patch replaces all occurrences of the default Priority.newInstance(0) with Priority.UNDEFINED. was (Author: seanpo03): YARN-5384.v10.patch replaces all occurences of the default Priority.newInstance(0) with Priority.UNDEFINED. > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch, YARN-5384.v10.patch, > YARN-5384.v2.patch, YARN-5384.v3.patch, YARN-5384.v4.patch, > YARN-5384.v5.patch, YARN-5384.v6.patch, YARN-5384.v7.patch, > YARN-5384.v8.patch, YARN-5384.v9.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5384: -- Attachment: YARN-5384.v10.patch YARN-5384.v10.patch replaces all occurences of the default Priority.newInstance(0) with Priority.UNDEFINED. > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch, YARN-5384.v10.patch, > YARN-5384.v2.patch, YARN-5384.v3.patch, YARN-5384.v4.patch, > YARN-5384.v5.patch, YARN-5384.v6.patch, YARN-5384.v7.patch, > YARN-5384.v8.patch, YARN-5384.v9.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5384: -- Attachment: YARN-5384.v9.patch YARN-5384.v9.patch addresses all your comments [~subru]. Thanks! > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch, YARN-5384.v2.patch, > YARN-5384.v3.patch, YARN-5384.v4.patch, YARN-5384.v5.patch, > YARN-5384.v6.patch, YARN-5384.v7.patch, YARN-5384.v8.patch, YARN-5384.v9.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5384: -- Attachment: YARN-5384.v8.patch YARN-5384.v8.patch removes checkstyle issues introduced by YARN-5384.v7.patch. > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch, YARN-5384.v2.patch, > YARN-5384.v3.patch, YARN-5384.v4.patch, YARN-5384.v5.patch, > YARN-5384.v6.patch, YARN-5384.v7.patch, YARN-5384.v8.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5384: -- Attachment: YARN-5384.v7.patch YARN-5384.v7.patch fixes the test failures in TestGreedyReservationAgent.java. > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch, YARN-5384.v2.patch, > YARN-5384.v3.patch, YARN-5384.v4.patch, YARN-5384.v5.patch, > YARN-5384.v6.patch, YARN-5384.v7.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5384: -- Attachment: YARN-5384.v6.patch YARN-5384.v6.patch fixes the checkstyle issues. > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch, YARN-5384.v2.patch, > YARN-5384.v3.patch, YARN-5384.v4.patch, YARN-5384.v5.patch, YARN-5384.v6.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5384: -- Attachment: YARN-5384.v5.patch Thanks [~subru] for your comments! YARN-5384.v5.patch addresses all your comments, and removes some unrelated changes from YARN-5384.v4.patch, > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch, YARN-5384.v2.patch, > YARN-5384.v3.patch, YARN-5384.v4.patch, YARN-5384.v5.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5384: -- Attachment: YARN-5384.v4.patch > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch, YARN-5384.v2.patch, > YARN-5384.v3.patch, YARN-5384.v4.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5384: -- Attachment: YARN-5384.v3.patch YARN-5384.v3.patch fixes the checkstyle issue with an unused import. However, ReservationDefinition.java does not have whitespace at line 184, nor was it introduced with YARN-5384.v2.patch of the patch so the issue is ignored for now. > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch, YARN-5384.v2.patch, > YARN-5384.v3.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5385) Add a PriorityAgent in ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5385: -- Attachment: YARN-5385.v1.patch > Add a PriorityAgent in ReservationSystem > - > > Key: YARN-5385 > URL: https://issues.apache.org/jira/browse/YARN-5385 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5385.v1.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the addition > of a priority agent to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5384) Expose priority in ReservationSystem submission APIs
[ https://issues.apache.org/jira/browse/YARN-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5384: -- Attachment: YARN-5384.v2.patch YARN-5384.v2.patch fixes the merge conflicts with latest trunk. > Expose priority in ReservationSystem submission APIs > > > Key: YARN-5384 > URL: https://issues.apache.org/jira/browse/YARN-5384 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5384.v1.patch, YARN-5384.v2.patch > > > YARN-5211 proposes adding support for generalized priorities for reservations > in the YARN ReservationSystem. This JIRA is a sub-task to track the changes > needed in ApplicationClientProtocol to accomplish it. Please refer to the > design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5656) Fix ReservationACLsTestBase
[ https://issues.apache.org/jira/browse/YARN-5656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507677#comment-15507677 ] Sean Po commented on YARN-5656: --- Thanks for the review and commit [~asuresh]! > Fix ReservationACLsTestBase > --- > > Key: YARN-5656 > URL: https://issues.apache.org/jira/browse/YARN-5656 > Project: Hadoop YARN > Issue Type: Test >Reporter: Sean Po >Assignee: Sean Po > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5656.v1.patch, YARN-5656.v2.patch > > > ReservationACLsTestBase fails when verifying that a reservation can be > successfully updated by a user who did not submit the reservation who also > has an admin ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5656) ReservationACLsTestBase fails on trunk
[ https://issues.apache.org/jira/browse/YARN-5656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507475#comment-15507475 ] Sean Po commented on YARN-5656: --- The Javadoc failures are not caused by the latest patch. Changes were made to only four files: NoOverCommitPolicy.java, [MismatchedUserException.java], ReservationACLsTestBase.java and TestNoOverCommitPolicy.java. None of these were referenced in the Javadoc failure results. > ReservationACLsTestBase fails on trunk > -- > > Key: YARN-5656 > URL: https://issues.apache.org/jira/browse/YARN-5656 > Project: Hadoop YARN > Issue Type: Test >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5656.v1.patch, YARN-5656.v2.patch > > > ReservationACLsTestBase fails when verifying that a reservation can be > successfully updated by a user who did not submit the reservation who also > has an admin ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5656) ReservationACLsTestBase fails on trunk
[ https://issues.apache.org/jira/browse/YARN-5656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5656: -- Attachment: YARN-5656.v2.patch Thanks [~asuresh] for the review. YARN-5656.v2.patch addresses your comments, and removes the unused MismatchedUserException.java file. > ReservationACLsTestBase fails on trunk > -- > > Key: YARN-5656 > URL: https://issues.apache.org/jira/browse/YARN-5656 > Project: Hadoop YARN > Issue Type: Test >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5656.v1.patch, YARN-5656.v2.patch > > > ReservationACLsTestBase fails when verifying that a reservation can be > successfully updated by a user who did not submit the reservation who also > has an admin ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5656) ReservationACLsTestBase fails on trunk
[ https://issues.apache.org/jira/browse/YARN-5656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497412#comment-15497412 ] Sean Po commented on YARN-5656: --- I think it is safe to remove the check entirely given the two scenarios: 1. If ACLS are enabled, then a check that ensures that the reservation is being updated by an admin / by the user who submitted the reservation is redundant. 2. Otherwise, if ACLs are not enabled, we are saying that anyone can update any reservation, so the check would be invalid. > ReservationACLsTestBase fails on trunk > -- > > Key: YARN-5656 > URL: https://issues.apache.org/jira/browse/YARN-5656 > Project: Hadoop YARN > Issue Type: Test >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5656.v1.patch > > > ReservationACLsTestBase fails when verifying that a reservation can be > successfully updated by a user who did not submit the reservation who also > has an admin ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5656) ReservationACLsTestBase fails on trunk
[ https://issues.apache.org/jira/browse/YARN-5656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-5656: -- Attachment: YARN-5656.v1.patch YARN-5656.v1.patch removes the check to verify that the same user is editing the reservation in the NoOverCommitPolicy. The patch also specifies the DominantResourceFairness policy in the ReservationACLsTestBase. > ReservationACLsTestBase fails on trunk > -- > > Key: YARN-5656 > URL: https://issues.apache.org/jira/browse/YARN-5656 > Project: Hadoop YARN > Issue Type: Test >Reporter: Sean Po > Attachments: YARN-5656.v1.patch > > > ReservationACLsTestBase fails when verifying that a reservation can be > successfully updated by a user who did not submit the reservation who also > has an admin ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5656) ReservationACLsTestBase fails on trunk
Sean Po created YARN-5656: - Summary: ReservationACLsTestBase fails on trunk Key: YARN-5656 URL: https://issues.apache.org/jira/browse/YARN-5656 Project: Hadoop YARN Issue Type: Test Reporter: Sean Po ReservationACLsTestBase fails when verifying that a reservation can be successfully updated by a user who did not submit the reservation who also has an admin ACL. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5331) Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468984#comment-15468984 ] Sean Po commented on YARN-5331: --- Thanks for the quick update [~ajsangeetha]! A few more minor comments: * NIT: PeriodicRLESparseResourceAllocation.java+32 ** Please put a javadoc for the constructor, and make it clear that timePeriod is in ms. * PeriodicRLESparseResourceAllocation.java+68 ** I noticed you changed the comparison operator from <= to <. I think <= is actually correct. The reason is because addInterval seems to set the tick at endTime = ZERO_RESOURCE before adding to existing RLE. *Currently*, if interval.getEndTime() == timePeriod - 1 (which is the latest acceptable endTime), then RLE at tick timePeriod -1 will be set to ZERO_RESOURCE. I believe this is a bug. ** Please add a unit test to verify this logic. A simple case would be to setCapacityInInterval with interval from 0 to the latest acceptable endTime, and make sure that getCapacityAtTime(timePeriod - 1) is non zero. I believe this test case will fail with current code. * NIT: PeriodicRLESparseResourceAllocation.java+83 ** Similar checks from PeriodicRLESparseResourceAllocation.java+68 should be made here. * TestPeriodicRLESparseResourceAllocation.java ** Since we have checks in PeriodicRLESparseResourceAllocation::setCapacityInInterval and PeriodicRLESparseResourceAllocation::removeInterval for valid time intervals, lets add positive and negative unit tests for these. > Extend RLESparseResourceAllocation with period for supporting recurring > reservations in YARN ReservationSystem > -- > > Key: YARN-5331 > URL: https://issues.apache.org/jira/browse/YARN-5331 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Sangeetha Abdu Jyothi > Attachments: YARN-5331.001.patch, YARN-5331.002.patch, > YARN-5331.003.patch > > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to add a > PeriodicRLESparseResourceAllocation. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5331) Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468705#comment-15468705 ] Sean Po commented on YARN-5331: --- Thanks [~ajsangeetha] for the patch! This looks good, except for a few very minor comments: * NIT: PeriodicRLESParseResourceAllocation.java+27 ** Remove extra newline. * NIT: TestPeriodicRLESparseResourceAllocation.java+18 ** Add a newline. * NIT: PeriodicRLESParseResourceAllocation.java+57 ** It looks like RLESparseResourceAllocation.AddInterval would include 0, and does not include interval.getEndTime(). Please update the javadoc. * NIT: PeriodicRLESParseResourceAllocation.java+61 ** Extra space before "specified". * PeriodicRLESParseResourceAllocation.java+65 ** Do we need to check interval.getStartTime() >= 0 and interval.getEndTime() > interval.getStartTime()? * TestPeriodicRLESparseResourceAllocation.java+81 ** Can be static. ** Also looks a lot like ReservationSystemTestUtil::generateAllocation. Maybe you can use that method instead, or move this one into ReservationSystemTestUtil. That way you can re-use in TestRLESparseResourceAllocation as well. * TestRLESparseResourceAllocation.java+528 ** Please include tests for this where tick > lastKey. Is it expected to be ZERO_RESOURCE? ** Also include test for tick == lastKey. Question about PeriodicRLESparseResourceAllocation::setTimePeriod: * If the new period is smaller than the previous period, should the behavior be to trim rleVector? It looks like there will be a bug for PeriodicRLESparseResourceAllocation::getMaxPeriodicCapacity if rleVector isn't trimmed, because the end condition is t <= lastKey. ** Please add a unit test for this condition. > Extend RLESparseResourceAllocation with period for supporting recurring > reservations in YARN ReservationSystem > -- > > Key: YARN-5331 > URL: https://issues.apache.org/jira/browse/YARN-5331 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Sangeetha Abdu Jyothi > Attachments: YARN-5331.001.patch, YARN-5331.002.patch > > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to add a > PeriodicRLESparseResourceAllocation. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5560) Clean up bad exception catching practices in TestYarnClient
[
https://issues.apache.org/jira/browse/YARN-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15446434#comment-15446434
]
Sean Po commented on YARN-5560:
---
Thanks [~jlowe] for the review and committing!
> Clean up bad exception catching practices in TestYarnClient
> ---
>
> Key: YARN-5560
> URL: https://issues.apache.org/jira/browse/YARN-5560
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Sean Po
>Assignee: Sean Po
> Fix For: 2.8.0
>
> Attachments: YARN-5560.v1.patch, YARN-5560.v2.patch,
> YARN-5560.v3.patch
>
>
> In TestYarnClient, tests commonly wrap methods that throw exceptions in a try
> catch statement similar to the following:
> {code}
> try {
> client.submitApplication(context);
> } catch (Exception e) {
> Assert.fail("Exception is not expected.");
> }
> {code}
> This hides useful error messages, and surfaces less helpful ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5560) Clean up bad exception catching practices in TestYarnClient
[
https://issues.apache.org/jira/browse/YARN-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Po updated YARN-5560:
--
Attachment: YARN-5560.v3.patch
You're welcome [~jlowe], thanks for reviewing! V3 of the patch addresses your
previous comment.
> Clean up bad exception catching practices in TestYarnClient
> ---
>
> Key: YARN-5560
> URL: https://issues.apache.org/jira/browse/YARN-5560
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Sean Po
>Assignee: Sean Po
> Attachments: YARN-5560.v1.patch, YARN-5560.v2.patch,
> YARN-5560.v3.patch
>
>
> In TestYarnClient, tests commonly wrap methods that throw exceptions in a try
> catch statement similar to the following:
> {code}
> try {
> client.submitApplication(context);
> } catch (Exception e) {
> Assert.fail("Exception is not expected.");
> }
> {code}
> This hides useful error messages, and surfaces less helpful ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5560) Clean up bad exception catching practices in TestYarnClient
[
https://issues.apache.org/jira/browse/YARN-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Po updated YARN-5560:
--
Attachment: YARN-5560.v2.patch
V2 of the patch fixes the open checkstyle issue.
> Clean up bad exception catching practices in TestYarnClient
> ---
>
> Key: YARN-5560
> URL: https://issues.apache.org/jira/browse/YARN-5560
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Sean Po
>Assignee: Sean Po
> Attachments: YARN-5560.v1.patch, YARN-5560.v2.patch
>
>
> In TestYarnClient, tests commonly wrap methods that throw exceptions in a try
> catch statement similar to the following:
> {code}
> try {
> client.submitApplication(context);
> } catch (Exception e) {
> Assert.fail("Exception is not expected.");
> }
> {code}
> This hides useful error messages, and surfaces less helpful ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5560) Clean up bad exception catching practices in TestYarnClient
[
https://issues.apache.org/jira/browse/YARN-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Po updated YARN-5560:
--
Attachment: YARN-5560.v1.patch
First patch removes occurrences of catch blocks that only invoke Assert.fail.
> Clean up bad exception catching practices in TestYarnClient
> ---
>
> Key: YARN-5560
> URL: https://issues.apache.org/jira/browse/YARN-5560
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Sean Po
>Assignee: Sean Po
> Attachments: YARN-5560.v1.patch
>
>
> In TestYarnClient, tests commonly wrap methods that throw exceptions in a try
> catch statement similar to the following:
> {code}
> try {
> client.submitApplication(context);
> } catch (Exception e) {
> Assert.fail("Exception is not expected.");
> }
> {code}
> This hides useful error messages, and surfaces less helpful ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5560) Clean up bad exception catching practices in TestYarnClient
[
https://issues.apache.org/jira/browse/YARN-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437460#comment-15437460
]
Sean Po edited comment on YARN-5560 at 8/25/16 6:59 PM:
First patch removes occurrences of catch blocks that only invoke Assert.fail in
TestYarnClient.
was (Author: seanpo03):
First patch removes occurrences of catch blocks that only invoke Assert.fail.
> Clean up bad exception catching practices in TestYarnClient
> ---
>
> Key: YARN-5560
> URL: https://issues.apache.org/jira/browse/YARN-5560
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Sean Po
>Assignee: Sean Po
> Attachments: YARN-5560.v1.patch
>
>
> In TestYarnClient, tests commonly wrap methods that throw exceptions in a try
> catch statement similar to the following:
> {code}
> try {
> client.submitApplication(context);
> } catch (Exception e) {
> Assert.fail("Exception is not expected.");
> }
> {code}
> This hides useful error messages, and surfaces less helpful ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5389) TestYarnClient#testReservationDelete fails
[
https://issues.apache.org/jira/browse/YARN-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437252#comment-15437252
]
Sean Po commented on YARN-5389:
---
Thanks for the review [~jlowe]!
> TestYarnClient#testReservationDelete fails
> --
>
> Key: YARN-5389
> URL: https://issues.apache.org/jira/browse/YARN-5389
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Rohith Sharma K S
>Assignee: Sean Po
> Labels: test
> Fix For: 2.8.0
>
> Attachments: YARN-5389.v1.patch, YARN-5389.v2.patch,
> YARN-5389.v3.patch, YARN-5389.v4.patch
>
>
> In build report
> [report|https://builds.apache.org/job/PreCommit-YARN-Build/12341/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt],
> below test fails.
> {noformat}
> Tests run: 28, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 26.066 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.client.api.impl.TestYarnClient
> testReservationDelete(org.apache.hadoop.yarn.client.api.impl.TestYarnClient)
> Time elapsed: 2.213 sec <<< FAILURE!
> java.lang.AssertionError: Exhausted attempts in checking if node capacity was
> added to the plan
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testReservationDelete(TestYarnClient.java:1584)
> testListReservationsByInvalidTimeInterval(org.apache.hadoop.yarn.client.api.impl.TestYarnClient)
> Time elapsed: 2.215 sec <<< FAILURE!
> java.lang.AssertionError: Exhausted attempts in checking if node capacity was
> added to the plan
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testListReservationsByInvalidTimeInterval(TestYarnClient.java:1444)
> testListReservationsByTimeIntervalContainingNoReservations(org.apache.hadoop.yarn.client.api.impl.TestYarnClient)
> Time elapsed: 2.206 sec <<< FAILURE!
> java.lang.AssertionError: Exhausted attempts in checking if node capacity was
> added to the plan
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testListReservationsByTimeIntervalContainingNoReservations(TestYarnClient.java:1494)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5389) TestYarnClient#testReservationDelete fails in trunk
[
https://issues.apache.org/jira/browse/YARN-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Po updated YARN-5389:
--
Attachment: YARN-5389.v4.patch
YARN-5389.v4.patch contains the changes spoken about in my previous comment.
> TestYarnClient#testReservationDelete fails in trunk
> ---
>
> Key: YARN-5389
> URL: https://issues.apache.org/jira/browse/YARN-5389
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Rohith Sharma K S
>Assignee: Sean Po
> Labels: test
> Attachments: YARN-5389.v1.patch, YARN-5389.v2.patch,
> YARN-5389.v3.patch, YARN-5389.v4.patch
>
>
> In build report
> [report|https://builds.apache.org/job/PreCommit-YARN-Build/12341/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt],
> below test fails.
> {noformat}
> Tests run: 28, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 26.066 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.client.api.impl.TestYarnClient
> testReservationDelete(org.apache.hadoop.yarn.client.api.impl.TestYarnClient)
> Time elapsed: 2.213 sec <<< FAILURE!
> java.lang.AssertionError: Exhausted attempts in checking if node capacity was
> added to the plan
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testReservationDelete(TestYarnClient.java:1584)
> testListReservationsByInvalidTimeInterval(org.apache.hadoop.yarn.client.api.impl.TestYarnClient)
> Time elapsed: 2.215 sec <<< FAILURE!
> java.lang.AssertionError: Exhausted attempts in checking if node capacity was
> added to the plan
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testListReservationsByInvalidTimeInterval(TestYarnClient.java:1444)
> testListReservationsByTimeIntervalContainingNoReservations(org.apache.hadoop.yarn.client.api.impl.TestYarnClient)
> Time elapsed: 2.206 sec <<< FAILURE!
> java.lang.AssertionError: Exhausted attempts in checking if node capacity was
> added to the plan
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testListReservationsByTimeIntervalContainingNoReservations(TestYarnClient.java:1494)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5389) TestYarnClient#testReservationDelete fails in trunk
[
https://issues.apache.org/jira/browse/YARN-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435907#comment-15435907
]
Sean Po commented on YARN-5389:
---
Thanks for the comments [~jlowe]. I agree - in the next patch, I will revert
changes that remove code that does something similar to the following:
{code}
try {
sResponse = client.submitReservation(sRequest);
} catch (Exception e) {
Assert.fail(e.getMessage());
}
{code}
I've created YARN-5560 to track cleaning up such patterns in TestYarnClient.
> TestYarnClient#testReservationDelete fails in trunk
> ---
>
> Key: YARN-5389
> URL: https://issues.apache.org/jira/browse/YARN-5389
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Rohith Sharma K S
>Assignee: Sean Po
> Labels: test
> Attachments: YARN-5389.v1.patch, YARN-5389.v2.patch,
> YARN-5389.v3.patch
>
>
> In build report
> [report|https://builds.apache.org/job/PreCommit-YARN-Build/12341/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt],
> below test fails.
> {noformat}
> Tests run: 28, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 26.066 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.client.api.impl.TestYarnClient
> testReservationDelete(org.apache.hadoop.yarn.client.api.impl.TestYarnClient)
> Time elapsed: 2.213 sec <<< FAILURE!
> java.lang.AssertionError: Exhausted attempts in checking if node capacity was
> added to the plan
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testReservationDelete(TestYarnClient.java:1584)
> testListReservationsByInvalidTimeInterval(org.apache.hadoop.yarn.client.api.impl.TestYarnClient)
> Time elapsed: 2.215 sec <<< FAILURE!
> java.lang.AssertionError: Exhausted attempts in checking if node capacity was
> added to the plan
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testListReservationsByInvalidTimeInterval(TestYarnClient.java:1444)
> testListReservationsByTimeIntervalContainingNoReservations(org.apache.hadoop.yarn.client.api.impl.TestYarnClient)
> Time elapsed: 2.206 sec <<< FAILURE!
> java.lang.AssertionError: Exhausted attempts in checking if node capacity was
> added to the plan
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testListReservationsByTimeIntervalContainingNoReservations(TestYarnClient.java:1494)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5560) Clean up bad exception catching practices in TestYarnClient
[
https://issues.apache.org/jira/browse/YARN-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Po updated YARN-5560:
--
Description:
In TestYarnClient, tests commonly wrap methods that throw exceptions in a try
catch statement similar to the following:
{code}
try {
client.submitApplication(context);
} catch (Exception e) {
Assert.fail("Exception is not expected.");
}
{code}
This hides useful error messages, and surfaces less helpful ones.
was:
In TestYarnClient, tests commonly wrap methods that throw exceptions in a try
catch statement similar to the following:
{{try {}}
{{client.submitApplication(context);}}
{{} catch (Exception e) {}}
{{Assert.fail("Exception is not expected.");}}
{{}}}
This hides useful error messages, and surfaces less helpful ones.
> Clean up bad exception catching practices in TestYarnClient
> ---
>
> Key: YARN-5560
> URL: https://issues.apache.org/jira/browse/YARN-5560
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Sean Po
>Assignee: Sean Po
>
> In TestYarnClient, tests commonly wrap methods that throw exceptions in a try
> catch statement similar to the following:
> {code}
> try {
> client.submitApplication(context);
> } catch (Exception e) {
> Assert.fail("Exception is not expected.");
> }
> {code}
> This hides useful error messages, and surfaces less helpful ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5560) Clean up bad exception catching practices in TestYarnClient
Sean Po created YARN-5560:
-
Summary: Clean up bad exception catching practices in
TestYarnClient
Key: YARN-5560
URL: https://issues.apache.org/jira/browse/YARN-5560
Project: Hadoop YARN
Issue Type: Test
Reporter: Sean Po
Assignee: Sean Po
In TestYarnClient, tests commonly wrap methods that throw exceptions in a try
catch statement similar to the following:
{{try {}}
{{client.submitApplication(context);}}
{{} catch (Exception e) {}}
{{Assert.fail("Exception is not expected.");}}
{{}}}
This hides useful error messages, and surfaces less helpful ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5389) TestYarnClient#testReservationDelete fails in trunk
[
https://issues.apache.org/jira/browse/YARN-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Po updated YARN-5389:
--
Attachment: YARN-5389.v3.patch
Thanks [~jlowe], the 10ms suggestion was overlooked in your previous
suggestion, but the change is made in v3 of the patch. I also removed any try
catch that asserts fail in the event of an exception to let the exception
bubble up.
> TestYarnClient#testReservationDelete fails in trunk
> ---
>
> Key: YARN-5389
> URL: https://issues.apache.org/jira/browse/YARN-5389
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Rohith Sharma K S
>Assignee: Sean Po
> Labels: test
> Attachments: YARN-5389.v1.patch, YARN-5389.v2.patch,
> YARN-5389.v3.patch
>
>
> In build report
> [report|https://builds.apache.org/job/PreCommit-YARN-Build/12341/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt],
> below test fails.
> {noformat}
> Tests run: 28, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 26.066 sec
> <<< FAILURE! - in org.apache.hadoop.yarn.client.api.impl.TestYarnClient
> testReservationDelete(org.apache.hadoop.yarn.client.api.impl.TestYarnClient)
> Time elapsed: 2.213 sec <<< FAILURE!
> java.lang.AssertionError: Exhausted attempts in checking if node capacity was
> added to the plan
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testReservationDelete(TestYarnClient.java:1584)
> testListReservationsByInvalidTimeInterval(org.apache.hadoop.yarn.client.api.impl.TestYarnClient)
> Time elapsed: 2.215 sec <<< FAILURE!
> java.lang.AssertionError: Exhausted attempts in checking if node capacity was
> added to the plan
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testListReservationsByInvalidTimeInterval(TestYarnClient.java:1444)
> testListReservationsByTimeIntervalContainingNoReservations(org.apache.hadoop.yarn.client.api.impl.TestYarnClient)
> Time elapsed: 2.206 sec <<< FAILURE!
> java.lang.AssertionError: Exhausted attempts in checking if node capacity was
> added to the plan
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testListReservationsByTimeIntervalContainingNoReservations(TestYarnClient.java:1494)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5327) API changes required to support recurring reservations in the YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433994#comment-15433994 ] Sean Po commented on YARN-5327: --- Thanks for addressing the comments [~ajsangeetha], this patch looks good to me. +1 after the checkstyle issues have been resolved. > API changes required to support recurring reservations in the YARN > ReservationSystem > > > Key: YARN-5327 > URL: https://issues.apache.org/jira/browse/YARN-5327 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Sangeetha Abdu Jyothi > Attachments: YARN-5327.001.patch, YARN-5327.002.patch, > YARN-5327.003.patch, YARN-5327.004.patch, YARN-5327.005.patch > > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to track the changes needed > in ApplicationClientProtocol to accomplish it. Please refer to the design doc > in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
