[jira] [Commented] (YARN-2693) Priority Label Manager in RM to manage application priority based on configuration
[ https://issues.apache.org/jira/browse/YARN-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367477#comment-14367477 ] Sunil G commented on YARN-2693: --- Thank you [~wangda] for sharing comments. As we move in the queue specific config inside scheduler.Queue, are we also taking ACLs back to scheduler (ACL wrt priority). Its better to control ACL from outside YarnAuthorizer and config only can be kept w.r.t scheduler. Pls share your thoughts. Regarding methods in ApplicationPrioirtyManager, it looks overall fine but I suggest we may need * getClusterApplicationPriorities (if its range, that can be sent back) Priority Label Manager in RM to manage application priority based on configuration -- Key: YARN-2693 URL: https://issues.apache.org/jira/browse/YARN-2693 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Sunil G Assignee: Sunil G Attachments: 0001-YARN-2693.patch, 0002-YARN-2693.patch, 0003-YARN-2693.patch, 0004-YARN-2693.patch, 0005-YARN-2693.patch, 0006-YARN-2693.patch Focus of this JIRA is to have a centralized service to handle priority labels. Support operations such as * Add/Delete priority label to a specified queue * Manage integer mapping associated with each priority label * Support managing default priority label of a given queue * Expose interface to RM to validate priority label TO have simplified interface, Priority Manager will support only configuration file in contrast with admin cli and REST. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2693) Priority Label Manager in RM to manage application priority based on configuration
[ https://issues.apache.org/jira/browse/YARN-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366225#comment-14366225 ] Wangda Tan commented on YARN-2693: -- Some overall suggestions: 1) Instead of ApplicationPriorityPerQueue, queue's priority related fields could store in scheduler.Queue directly (methods of scheduler.Queue, implementation for different scheduler could be various. Since we have different scheduler configuration). Benefits of doing this: - All other queue-specifc configurations are in schedulers' own configuration files, make application-priority fields for queue storing out-of queue means you have to sync it with queue's configuration when you do refreshQueues, etc. - Put it in scheduler.Queue can make scheduler changes easier (don't have to access ApplicationPriorityManager). 2) Methods of ApplicationPriorityManager: - Since we're discussing how to configure priority, I will review ApplicationPriorityManager implementations once we close design. - ClusterPriorities should be a range (If we start from zero, a maxPriority will be enough) - getApplicationPriorityFromQueue should not exist, all queue related methods should be in scheduler.Queue - isPriorityExistsInCluster maybe not existed, it should be something like accepted. - Can be reinitialize - Can convert between number/label Priority Label Manager in RM to manage application priority based on configuration -- Key: YARN-2693 URL: https://issues.apache.org/jira/browse/YARN-2693 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Sunil G Assignee: Sunil G Attachments: 0001-YARN-2693.patch, 0002-YARN-2693.patch, 0003-YARN-2693.patch, 0004-YARN-2693.patch, 0005-YARN-2693.patch, 0006-YARN-2693.patch Focus of this JIRA is to have a centralized service to handle priority labels. Support operations such as * Add/Delete priority label to a specified queue * Manage integer mapping associated with each priority label * Support managing default priority label of a given queue * Expose interface to RM to validate priority label TO have simplified interface, Priority Manager will support only configuration file in contrast with admin cli and REST. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2693) Priority Label Manager in RM to manage application priority based on configuration
[
https://issues.apache.org/jira/browse/YARN-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347172#comment-14347172
]
Hadoop QA commented on YARN-2693:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702487/0006-YARN-2693.patch
against trunk revision 3560180.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1152 javac
compiler warnings (more than the trunk's current 1151 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationpriority.TestApplicationPriorityManager
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6837//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6837//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6837//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6837//console
This message is automatically generated.
Priority Label Manager in RM to manage application priority based on
configuration
--
Key: YARN-2693
URL: https://issues.apache.org/jira/browse/YARN-2693
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-YARN-2693.patch, 0002-YARN-2693.patch,
0003-YARN-2693.patch, 0004-YARN-2693.patch, 0005-YARN-2693.patch,
0006-YARN-2693.patch
Focus of this JIRA is to have a centralized service to handle priority labels.
Support operations such as
* Add/Delete priority label to a specified queue
* Manage integer mapping associated with each priority label
* Support managing default priority label of a given queue
* Expose interface to RM to validate priority label
TO have simplified interface, Priority Manager will support only
configuration file in contrast with admin cli and REST.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2693) Priority Label Manager in RM to manage application priority based on configuration
[
https://issues.apache.org/jira/browse/YARN-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343404#comment-14343404
]
Hadoop QA commented on YARN-2693:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701883/0006-YARN-2693.patch
against trunk revision ca1c00b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1152 javac
compiler warnings (more than the trunk's current 209 warnings).
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
47 warning messages.
See
https://builds.apache.org/job/PreCommit-YARN-Build/6801//artifact/patchprocess/diffJavadocWarnings.txt
for details.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6801//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6801//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6801//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6801//console
This message is automatically generated.
Priority Label Manager in RM to manage application priority based on
configuration
--
Key: YARN-2693
URL: https://issues.apache.org/jira/browse/YARN-2693
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-YARN-2693.patch, 0002-YARN-2693.patch,
0003-YARN-2693.patch, 0004-YARN-2693.patch, 0005-YARN-2693.patch,
0006-YARN-2693.patch
Focus of this JIRA is to have a centralized service to handle priority labels.
Support operations such as
* Add/Delete priority label to a specified queue
* Manage integer mapping associated with each priority label
* Support managing default priority label of a given queue
* Expose interface to RM to validate priority label
TO have simplified interface, Priority Manager will support only
configuration file in contrast with admin cli and REST.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2693) Priority Label Manager in RM to manage application priority based on configuration
[
https://issues.apache.org/jira/browse/YARN-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343435#comment-14343435
]
Hadoop QA commented on YARN-2693:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701889/0006-YARN-2693.patch
against trunk revision ca1c00b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1158 javac
compiler warnings (more than the trunk's current 1157 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6802//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6802//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6802//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6802//console
This message is automatically generated.
Priority Label Manager in RM to manage application priority based on
configuration
--
Key: YARN-2693
URL: https://issues.apache.org/jira/browse/YARN-2693
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-YARN-2693.patch, 0002-YARN-2693.patch,
0003-YARN-2693.patch, 0004-YARN-2693.patch, 0005-YARN-2693.patch,
0006-YARN-2693.patch
Focus of this JIRA is to have a centralized service to handle priority labels.
Support operations such as
* Add/Delete priority label to a specified queue
* Manage integer mapping associated with each priority label
* Support managing default priority label of a given queue
* Expose interface to RM to validate priority label
TO have simplified interface, Priority Manager will support only
configuration file in contrast with admin cli and REST.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2693) Priority Label Manager in RM to manage application priority based on configuration
[
https://issues.apache.org/jira/browse/YARN-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340015#comment-14340015
]
Hadoop QA commented on YARN-2693:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701313/0006-YARN-2693.patch
against trunk revision 48c7ee7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1152 javac
compiler warnings (more than the trunk's current 1151 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6777//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6777//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6777//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6777//console
This message is automatically generated.
Priority Label Manager in RM to manage application priority based on
configuration
--
Key: YARN-2693
URL: https://issues.apache.org/jira/browse/YARN-2693
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-YARN-2693.patch, 0002-YARN-2693.patch,
0003-YARN-2693.patch, 0004-YARN-2693.patch, 0005-YARN-2693.patch,
0006-YARN-2693.patch
Focus of this JIRA is to have a centralized service to handle priority labels.
Support operations such as
* Add/Delete priority label to a specified queue
* Manage integer mapping associated with each priority label
* Support managing default priority label of a given queue
* Expose interface to RM to validate priority label
TO have simplified interface, Priority Manager will support only
configuration file in contrast with admin cli and REST.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2693) Priority Label Manager in RM to manage application priority based on configuration
[
https://issues.apache.org/jira/browse/YARN-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338914#comment-14338914
]
Hadoop QA commented on YARN-2693:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701122/0006-YARN-2693.patch
against trunk revision dce8b9c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1152 javac
compiler warnings (more than the trunk's current 1151 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6758//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6758//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6758//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6758//console
This message is automatically generated.
Priority Label Manager in RM to manage application priority based on
configuration
--
Key: YARN-2693
URL: https://issues.apache.org/jira/browse/YARN-2693
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Reporter: Sunil G
Assignee: Sunil G
Attachments: 0001-YARN-2693.patch, 0002-YARN-2693.patch,
0003-YARN-2693.patch, 0004-YARN-2693.patch, 0005-YARN-2693.patch,
0006-YARN-2693.patch
Focus of this JIRA is to have a centralized service to handle priority labels.
Support operations such as
* Add/Delete priority label to a specified queue
* Manage integer mapping associated with each priority label
* Support managing default priority label of a given queue
* Expose interface to RM to validate priority label
TO have simplified interface, Priority Manager will support only
configuration file in contrast with admin cli and REST.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
