[jira] [Commented] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[ https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14159037#comment-14159037 ] Hudson commented on YARN-1711: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #700 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/700/]) YARN-1711. Policy to enforce instantaneous and over-time quotas on user reservation. Contributed by Carlo Curino and Subru Krishnan. (cdouglas: rev b6df0dddcdafd7ec67c76ea92aea3ff3e94db247) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/SharingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/PlanningException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/TestNoOverCommitPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/NoOverCommitPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/MismatchedUserException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/ContractValidationException.java * YARN-1051-CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/TestCapacityOverTimePolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/ResourceOverCommitException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/PlanningQuotaException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709 -- Key: YARN-1711 URL: https://issues.apache.org/jira/browse/YARN-1711 Project: Hadoop YARN Issue Type: Sub-task Reporter: Carlo Curino Assignee: Carlo Curino Labels: reservations Attachments: YARN-1711.1.patch, YARN-1711.2.patch, YARN-1711.3.patch, YARN-1711.4.patch, YARN-1711.5.patch, YARN-1711.patch This JIRA tracks the development of a policy that enforces user quotas (a time-extension of the notion of capacity) in the inventory subsystem discussed in YARN-1709. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[ https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14159144#comment-14159144 ] Hudson commented on YARN-1711: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1891 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1891/]) YARN-1711. Policy to enforce instantaneous and over-time quotas on user reservation. Contributed by Carlo Curino and Subru Krishnan. (cdouglas: rev b6df0dddcdafd7ec67c76ea92aea3ff3e94db247) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/SharingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/TestCapacityOverTimePolicy.java * YARN-1051-CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/TestNoOverCommitPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/ResourceOverCommitException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/PlanningException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/PlanningQuotaException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/MismatchedUserException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/ContractValidationException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/NoOverCommitPolicy.java CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709 -- Key: YARN-1711 URL: https://issues.apache.org/jira/browse/YARN-1711 Project: Hadoop YARN Issue Type: Sub-task Reporter: Carlo Curino Assignee: Carlo Curino Labels: reservations Attachments: YARN-1711.1.patch, YARN-1711.2.patch, YARN-1711.3.patch, YARN-1711.4.patch, YARN-1711.5.patch, YARN-1711.patch This JIRA tracks the development of a policy that enforces user quotas (a time-extension of the notion of capacity) in the inventory subsystem discussed in YARN-1709. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[ https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14159168#comment-14159168 ] Hudson commented on YARN-1711: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1916 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1916/]) YARN-1711. Policy to enforce instantaneous and over-time quotas on user reservation. Contributed by Carlo Curino and Subru Krishnan. (cdouglas: rev b6df0dddcdafd7ec67c76ea92aea3ff3e94db247) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/PlanningQuotaException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/TestCapacityOverTimePolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java * YARN-1051-CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/ResourceOverCommitException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/NoOverCommitPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/MismatchedUserException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/TestNoOverCommitPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/ContractValidationException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/SharingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/PlanningException.java CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709 -- Key: YARN-1711 URL: https://issues.apache.org/jira/browse/YARN-1711 Project: Hadoop YARN Issue Type: Sub-task Reporter: Carlo Curino Assignee: Carlo Curino Labels: reservations Attachments: YARN-1711.1.patch, YARN-1711.2.patch, YARN-1711.3.patch, YARN-1711.4.patch, YARN-1711.5.patch, YARN-1711.patch This JIRA tracks the development of a policy that enforces user quotas (a time-extension of the notion of capacity) in the inventory subsystem discussed in YARN-1709. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[ https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14158766#comment-14158766 ] Hudson commented on YARN-1711: -- FAILURE: Integrated in Hadoop-trunk-Commit #6189 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6189/]) YARN-1711. Policy to enforce instantaneous and over-time quotas on user reservation. Contributed by Carlo Curino and Subru Krishnan. (cdouglas: rev b6df0dddcdafd7ec67c76ea92aea3ff3e94db247) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/TestCapacityOverTimePolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/PlanningException.java * YARN-1051-CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/MismatchedUserException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/SharingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/TestNoOverCommitPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/PlanningQuotaException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/NoOverCommitPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/ContractValidationException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/exceptions/ResourceOverCommitException.java CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709 -- Key: YARN-1711 URL: https://issues.apache.org/jira/browse/YARN-1711 Project: Hadoop YARN Issue Type: Sub-task Reporter: Carlo Curino Assignee: Carlo Curino Labels: reservations Attachments: YARN-1711.1.patch, YARN-1711.2.patch, YARN-1711.3.patch, YARN-1711.4.patch, YARN-1711.5.patch, YARN-1711.patch This JIRA tracks the development of a policy that enforces user quotas (a time-extension of the notion of capacity) in the inventory subsystem discussed in YARN-1709. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[
https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136186#comment-14136186
]
Hadoop QA commented on YARN-1711:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12669195/YARN-1711.5.patch
against trunk revision 8e5d671.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4977//console
This message is automatically generated.
CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
--
Key: YARN-1711
URL: https://issues.apache.org/jira/browse/YARN-1711
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Carlo Curino
Assignee: Carlo Curino
Labels: reservations
Attachments: YARN-1711.1.patch, YARN-1711.2.patch, YARN-1711.3.patch,
YARN-1711.4.patch, YARN-1711.5.patch, YARN-1711.patch
This JIRA tracks the development of a policy that enforces user quotas (a
time-extension of the notion of capacity) in the inventory subsystem
discussed in YARN-1709.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[ https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14134693#comment-14134693 ] Carlo Curino commented on YARN-1711: [~chris.douglas] thanks for the prompt and precise feedback. I addressed them in the updated patch (v4) as follows: * added comments and implemented nits as requested * got rid of excludeList altogether, as this is not necessary anymore given restructuring done in th e ReservationSystem (i.e., the exclusion list is now handled outside the scope of the policies which is cleaner/simpler) * improved tests, by using subclasses of exception wherever possible, and scoping them so that it is more telling that a failure is what we wanted to be. * subclasses of PlanningException give callers some indication of why the call did not succeed (e.g., enough for the tests), future smarter agents might require more detailed explanation (e.g., the JSON payload you mention) * used the (expected = SomeException.class) notation for tests. Regarding annotations I am using @Public @Unstable for the exceptions, as they can bubble up all the way to users, while @LimitedPrivate(yarn) @Unstable for the other classes. If anyone has better suggestions for the annotations please advise. CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709 -- Key: YARN-1711 URL: https://issues.apache.org/jira/browse/YARN-1711 Project: Hadoop YARN Issue Type: Sub-task Reporter: Carlo Curino Assignee: Carlo Curino Labels: reservations Attachments: YARN-1711.1.patch, YARN-1711.2.patch, YARN-1711.3.patch, YARN-1711.patch This JIRA tracks the development of a policy that enforces user quotas (a time-extension of the notion of capacity) in the inventory subsystem discussed in YARN-1709. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[
https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14134714#comment-14134714
]
Hadoop QA commented on YARN-1711:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668909/YARN-1711.4.patch
against trunk revision 8008f0e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4962//console
This message is automatically generated.
CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
--
Key: YARN-1711
URL: https://issues.apache.org/jira/browse/YARN-1711
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Carlo Curino
Assignee: Carlo Curino
Labels: reservations
Attachments: YARN-1711.1.patch, YARN-1711.2.patch, YARN-1711.3.patch,
YARN-1711.4.patch, YARN-1711.patch
This JIRA tracks the development of a policy that enforces user quotas (a
time-extension of the notion of capacity) in the inventory subsystem
discussed in YARN-1709.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[ https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14134718#comment-14134718 ] Chris Douglas commented on YARN-1711: - +1 Thanks for addressing the feedback on the patch CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709 -- Key: YARN-1711 URL: https://issues.apache.org/jira/browse/YARN-1711 Project: Hadoop YARN Issue Type: Sub-task Reporter: Carlo Curino Assignee: Carlo Curino Labels: reservations Attachments: YARN-1711.1.patch, YARN-1711.2.patch, YARN-1711.3.patch, YARN-1711.4.patch, YARN-1711.patch This JIRA tracks the development of a policy that enforces user quotas (a time-extension of the notion of capacity) in the inventory subsystem discussed in YARN-1709. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[
https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14133406#comment-14133406
]
Chris Douglas commented on YARN-1711:
-
General
- The public classes should be annotated with the correct visibility and
stability annotations (probably {{@Public}} and {{@Unstable}})
{{SharingPolicy}}
- Javadoc throws clause and some parameters not populated
- In particular, the {{excludeList}} parameter could use some unpacking.
{{CapacityOverTimePolicy}}
- Just performing the cast, and throwing {{ClassCastException}} implicitly is
equally clear
- nit: spacing/concat: {{plan.getTotalCapacity() + )by + accepting
reservation: }}
- Just as an observation, no change requested: the assumption that the sharing
policy holds the lock on the plan is probably OK, but since both are interfaces
there may be a missing abstraction that associates compatible sets of
interlocking components.
- I can't think of a more appropriate solution to handling aggregates of
{{Resource}}. Anything more correct doesn't really justify the complexity,
certainly not before we get some more experience with planning. Since enabling
this is optional, enforcement with the {{IntegralResource}} is a pragmatic
tradeoff.
{{*Exception}}
- s/MismatchingUserException/MismatchedUserException/
- Are the subclasses of {{PlanningException}} for a caller to distinguish the
cause for the rejected request, so it can refine it? If that's the case, should
they contain diagnostic information as a payload e.g., requested vs actual
user? If the intent is to extract it, then some more easily parsed format for
the message might be appropriate (e.g., JSON).
{{NoOverCommitPolicy}}
- The {{excludeList}} should probably be final, and cleared/populated with a
clone of the set on calls to {{init()}}
{{CapacitySchedulerConfiguration}}
- Missing javadoc for the new parameters.
{{TestNoOverCommitPolicy}}
- Consider using {{@Test(expected = SomeException.class)}} instead of
{{Assert::fail()}} and try/catch for {{testSingleFail()}}
- Consider specifying the expected cause/subtype instead of generic
{{PlanningException}}
- {{testMultiTenantFail}} only veries that a {{PlanningException}} is thrown,
not that it fails as expected
{{TestCapacityOverTimePolicy}}
- Most of the tests don't verify that the failure occurs when and how its
parameters specify, but only check that a {{PlanningException}} is thrown.
CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
--
Key: YARN-1711
URL: https://issues.apache.org/jira/browse/YARN-1711
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Carlo Curino
Assignee: Carlo Curino
Labels: reservations
Attachments: YARN-1711.1.patch, YARN-1711.2.patch, YARN-1711.3.patch,
YARN-1711.patch
This JIRA tracks the development of a policy that enforces user quotas (a
time-extension of the notion of capacity) in the inventory subsystem
discussed in YARN-1709.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[
https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130945#comment-14130945
]
Hadoop QA commented on YARN-1711:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668207/YARN-1711.3.patch
against trunk revision 6c08339.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4901//console
This message is automatically generated.
CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
--
Key: YARN-1711
URL: https://issues.apache.org/jira/browse/YARN-1711
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Carlo Curino
Assignee: Carlo Curino
Labels: reservations
Attachments: YARN-1711.1.patch, YARN-1711.2.patch, YARN-1711.3.patch,
YARN-1711.patch
This JIRA tracks the development of a policy that enforces user quotas (a
time-extension of the notion of capacity) in the inventory subsystem
discussed in YARN-1709.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[
https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14127929#comment-14127929
]
Hadoop QA commented on YARN-1711:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667567/YARN-1712.3.patch
against trunk revision 0de563a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4867//console
This message is automatically generated.
CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
--
Key: YARN-1711
URL: https://issues.apache.org/jira/browse/YARN-1711
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Carlo Curino
Assignee: Carlo Curino
Labels: reservations
Attachments: YARN-1711.1.patch, YARN-1711.2.patch, YARN-1711.patch
This JIRA tracks the development of a policy that enforces user quotas (a
time-extension of the notion of capacity) in the inventory subsystem
discussed in YARN-1709.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[
https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14127941#comment-14127941
]
Hadoop QA commented on YARN-1711:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667570/YARN-1711.2.patch
against trunk revision 0de563a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4868//console
This message is automatically generated.
CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
--
Key: YARN-1711
URL: https://issues.apache.org/jira/browse/YARN-1711
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Carlo Curino
Assignee: Carlo Curino
Labels: reservations
Attachments: YARN-1711.1.patch, YARN-1711.2.patch, YARN-1711.patch
This JIRA tracks the development of a policy that enforces user quotas (a
time-extension of the notion of capacity) in the inventory subsystem
discussed in YARN-1709.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[
https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14116079#comment-14116079
]
Hadoop QA commented on YARN-1711:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665488/YARN-1711.1.patch
against trunk revision 9ad413b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4783//console
This message is automatically generated.
CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
--
Key: YARN-1711
URL: https://issues.apache.org/jira/browse/YARN-1711
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Carlo Curino
Assignee: Carlo Curino
Labels: reservations
Attachments: YARN-1711.1.patch, YARN-1711.patch
This JIRA tracks the development of a policy that enforces user quotas (a
time-extension of the notion of capacity) in the inventory subsystem
discussed in YARN-1709.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[ https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14045429#comment-14045429 ] Carlo Curino commented on YARN-1711: Attached a patch that provide 2 simple policies for sharing resources. * The NoOverCommitPolicy simply enforce plan physical resource availability (as multiple agents from YARN-1710 could concurrently try to add to the plan). * The CapacityOvertimePolicy is a natural exension of the notion of Capacity where each user is given a maximum instantaneous limit (e.g., 50% of the plan capacity in any instant in time), and a maximum integral limit (e.g., no more than 10% in average an any 24h period of time). Setting the 3 parameters of maxInstantaneous, maxAverage, and validationWindow, allows administrators to choose how flexible or strict is the sharing on one extreme the administrator can impose instantaneous limits equivalent to user-limit in a regular queue, and on the opposite it can allow users to grab large blocks of resources for large period of time (before having to yield to others). As mentioned for other sub-tasks of YARN-1051 the patch is marked available to signal its availability for review, but not committable on its own. CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709 -- Key: YARN-1711 URL: https://issues.apache.org/jira/browse/YARN-1711 Project: Hadoop YARN Issue Type: Sub-task Reporter: Carlo Curino Assignee: Carlo Curino Labels: reservations Attachments: YARN-1711.patch This JIRA tracks the development of a policy that enforces user quotas (a time-extension of the notion of capacity) in the inventory subsystem discussed in YARN-1709. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1711) CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
[
https://issues.apache.org/jira/browse/YARN-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14045438#comment-14045438
]
Hadoop QA commented on YARN-1711:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652734/YARN-1711.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4108//console
This message is automatically generated.
CapacityOverTimePolicy: a policy to enforce quotas over time for YARN-1709
--
Key: YARN-1711
URL: https://issues.apache.org/jira/browse/YARN-1711
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Carlo Curino
Assignee: Carlo Curino
Labels: reservations
Attachments: YARN-1711.patch
This JIRA tracks the development of a policy that enforces user quotas (a
time-extension of the notion of capacity) in the inventory subsystem
discussed in YARN-1709.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
