[jira] [Commented] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
[
https://issues.apache.org/jira/browse/YARN-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199559#comment-14199559
]
Gordon Wang commented on YARN-2808:
---
setting yarn.resourcemanager.max-completed-applications to 0 is not a solution
from my view.
RM could cache completed app generic info, so that it is easy for user to check
the job status. And more, user could find the job info in RM's web UI, then,
redirect to history-server/timeline-server for more app detail info.
I think YARN-1794 is similar, but not the same.
{quote}
And was intending to fix by modifing YarnClientImpl.getContainers(). If the
size of containers is 0. We can get it from timeline.
{quote}
I am afraid this fix is only for YARN-2808 case. But generally speaking, since
RM does not cache completed container info, if a app is running, both the size
of running containers and the size of completed container could be larger than
0. So, one possible fix is that pulling the container info both from RM and
timeline-server regardless of the state of app attempt.
yarn client tool can not list app_attempt's container info correctly
Key: YARN-2808
URL: https://issues.apache.org/jira/browse/YARN-2808
Project: Hadoop YARN
Issue Type: Bug
Components: client
Reporter: Gordon Wang
When enabling timeline server, yarn client can not list the container info
for a application attempt correctly.
Here is the reproduce step.
# enabling yarn timeline server
# submit a MR job
# after the job is finished. use yarn client to list the container info of
the app attempt.
Then, since the RM has cached the application's attempt info, the output show
{noformat}
[hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
appattempt_1415168250217_0001_01
14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
http://0.0.0.0:8188/ws/v1/timeline/
14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
server at /0.0.0.0:10200
Total number of containers :0
Container-Id Start Time Finish
Time StateHost
LOG-URL
{noformat}
But if the rm is restarted, client can fetch the container info from timeline
server correctly.
{noformat}
[hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
appattempt_1415168250217_0001_01
14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
http://0.0.0.0:8188/ws/v1/timeline/
14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
server at /0.0.0.0:10200
Total number of containers :4
Container-Id Start Time Finish
Time StateHost
LOG-URL
container_1415168250217_0001_01_01 1415168318376
1415168349896COMPLETElocalhost.localdomain:47024
http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
container_1415168250217_0001_01_02 1415168326399
1415168334858COMPLETElocalhost.localdomain:47024
http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
container_1415168250217_0001_01_03 1415168326400
1415168335277COMPLETElocalhost.localdomain:47024
http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
container_1415168250217_0001_01_04 1415168335825
1415168343873COMPLETElocalhost.localdomain:47024
http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-2808) yarn client tool can not list app_attempt's container info correctly
Gordon Wang created YARN-2808:
-
Summary: yarn client tool can not list app_attempt's container
info correctly
Key: YARN-2808
URL: https://issues.apache.org/jira/browse/YARN-2808
Project: Hadoop YARN
Issue Type: Bug
Components: client
Reporter: Gordon Wang
When enabling timeline server, yarn client can not list the container info for
a application attempt correctly.
Here is the reproduce step.
# enabling yarn timeline server
# submit a MR job
# after the job is finished. use yarn client to list the container info of the
app attempt.
Then, since the RM has cached the application's attempt info, the output show
{noformat}
[hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
appattempt_1415168250217_0001_01
14/11/05 01:19:15 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/11/05 01:19:15 INFO impl.TimelineClientImpl: Timeline service address:
http://0.0.0.0:8188/ws/v1/timeline/
14/11/05 01:19:16 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
14/11/05 01:19:16 INFO client.AHSProxy: Connecting to Application History
server at /0.0.0.0:10200
Total number of containers :0
Container-IdStart Time Finish Time
StateHost
LOG-URL
{noformat}
But if the rm is restarted, client can fetch the container info from timeline
server correctly.
{noformat}
[hadoop@localhost hadoop-3.0.0-SNAPSHOT]$ ./bin/yarn container -list
appattempt_1415168250217_0001_01
14/11/05 01:21:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/11/05 01:21:06 INFO impl.TimelineClientImpl: Timeline service address:
http://0.0.0.0:8188/ws/v1/timeline/
14/11/05 01:21:06 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
14/11/05 01:21:06 INFO client.AHSProxy: Connecting to Application History
server at /0.0.0.0:10200
Total number of containers :4
Container-IdStart Time Finish Time
StateHost
LOG-URL
container_1415168250217_0001_01_01 1415168318376
1415168349896COMPLETElocalhost.localdomain:47024
http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_01/container_1415168250217_0001_01_01/hadoop
container_1415168250217_0001_01_02 1415168326399
1415168334858COMPLETElocalhost.localdomain:47024
http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_02/container_1415168250217_0001_01_02/hadoop
container_1415168250217_0001_01_03 1415168326400
1415168335277COMPLETElocalhost.localdomain:47024
http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_03/container_1415168250217_0001_01_03/hadoop
container_1415168250217_0001_01_04 1415168335825
1415168343873COMPLETElocalhost.localdomain:47024
http://0.0.0.0:8188/applicationhistory/logs/localhost.localdomain:47024/container_1415168250217_0001_01_04/container_1415168250217_0001_01_04/hadoop
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-1950) Do not check ACL in Yarn scheduler when Yarn ACL is disabled
[ https://issues.apache.org/jira/browse/YARN-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gordon Wang updated YARN-1950: -- Issue Type: Sub-task (was: Bug) Parent: YARN-1941 Do not check ACL in Yarn scheduler when Yarn ACL is disabled Key: YARN-1950 URL: https://issues.apache.org/jira/browse/YARN-1950 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Affects Versions: 2.3.0 Reporter: Gordon Wang Assignee: Gordon Wang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1941) Yarn scheduler ACL improvement
[ https://issues.apache.org/jira/browse/YARN-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13982355#comment-13982355 ] Gordon Wang commented on YARN-1941: --- [~rusanu], from the dev engineer view, I can not agree with you more. But as [~sandyr] mentioned, if a lot of customers are using this, We should evaluate the impact of this change. I reconsider this. I prefer to change the root ACL now. And here is my thought. 1. If we change root ACL to , users who leave root ACL as default should have to update their scheduler configuration file. But it is an easy update, just implicitly set root ACL as * is OK. 2. Since most of the users may use ClouderaManager or Ambari to do the upgrade job. This kind of change can be easily implemented in these tools. Thus, the impact of users will be minimised. For the users who directly use hadoop binary, they must be the experienced hadoop admins. Change this configuration is not a hard job for them during cluster upgrade. [~sandyr], what are your opinions? could you please help me to involve more YARN guys to discuss this? Your comments are very welcome ! Thanks. Yarn scheduler ACL improvement -- Key: YARN-1941 URL: https://issues.apache.org/jira/browse/YARN-1941 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.3.0 Reporter: Gordon Wang Assignee: Gordon Wang Labels: scheduler Defect: 1. Currently, in Yarn Capacity Scheduler and Yarn Fair Scheduler, the queue ACL is always checked when submitting a app to scheduler, regardless of the property yarn.acl.enable. But for killing an app, the ACL is checked when yarn.acl.enable is set. The behaviour is not consistent. 2. default ACL for root queue is EVERYBODY_ACL( * ), while default ACL for other queues is NODODY_ACL( ). From users' view, this is error prone and not easy to understand the ACL policy of Yarn scheduler. root queue should not be so special compared with other parent queues. For example, if I want to set capacity scheduler ACL, the ACL of root has to be set explicitly. Otherwise, everyone can submit APP to yarn scheduler. Because root queue ACL is EVERYBODY_ACL. This is hard for user to administrate yarn scheduler. So, I propose to improve the ACL of yarn scheduler in the following aspects. 1. only enable scheduler queue ACL when yarn.acl.enable is set to true. 2. set the default ACL of root queue as NOBODY_ACL( ). Make all the parent queues' ACL consistent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (YARN-1950) Do not check ACL in Yarn scheduler when Yarn ACL is disabled
[ https://issues.apache.org/jira/browse/YARN-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gordon Wang reassigned YARN-1950: - Assignee: Gordon Wang Do not check ACL in Yarn scheduler when Yarn ACL is disabled Key: YARN-1950 URL: https://issues.apache.org/jira/browse/YARN-1950 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.3.0 Reporter: Gordon Wang Assignee: Gordon Wang -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1941) Yarn scheduler ACL improvement
[ https://issues.apache.org/jira/browse/YARN-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13973662#comment-13973662 ] Gordon Wang commented on YARN-1941: --- [~sandyr], I agree with you. Your use case is legitimate. Let's keep the default ACL of root as EVERYBODY_ACL. Yarn scheduler ACL improvement -- Key: YARN-1941 URL: https://issues.apache.org/jira/browse/YARN-1941 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.3.0 Reporter: Gordon Wang Assignee: Gordon Wang Labels: scheduler Defect: 1. Currently, in Yarn Capacity Scheduler and Yarn Fair Scheduler, the queue ACL is always checked when submitting a app to scheduler, regardless of the property yarn.acl.enable. But for killing an app, the ACL is checked when yarn.acl.enable is set. The behaviour is not consistent. 2. default ACL for root queue is EVERYBODY_ACL( * ), while default ACL for other queues is NODODY_ACL( ). From users' view, this is error prone and not easy to understand the ACL policy of Yarn scheduler. root queue should not be so special compared with other parent queues. For example, if I want to set capacity scheduler ACL, the ACL of root has to be set explicitly. Otherwise, everyone can submit APP to yarn scheduler. Because root queue ACL is EVERYBODY_ACL. This is hard for user to administrate yarn scheduler. So, I propose to improve the ACL of yarn scheduler in the following aspects. 1. only enable scheduler queue ACL when yarn.acl.enable is set to true. 2. set the default ACL of root queue as NOBODY_ACL( ). Make all the parent queues' ACL consistent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (YARN-1951) Change root queue default ACL to NOBODY_ACL in yarn scheduler
[ https://issues.apache.org/jira/browse/YARN-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gordon Wang resolved YARN-1951. --- Resolution: Won't Fix Per discussion with [~sandyr] in YARN-1941. The cost of breaking existing setups outweighs the benefits of changing this. Let's keep the root queue default ACL. Change root queue default ACL to NOBODY_ACL in yarn scheduler - Key: YARN-1951 URL: https://issues.apache.org/jira/browse/YARN-1951 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Affects Versions: 2.3.0 Reporter: Gordon Wang -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1941) Yarn scheduler ACL improvement
[ https://issues.apache.org/jira/browse/YARN-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13971442#comment-13971442 ] Gordon Wang commented on YARN-1941: --- Hi [~sandyr], Thank you for your reply ! For (2), we can keep the behaviour compatible even if we change the default ACL to NOBODY_ACL. From the user's view, changing the default ACL will not need user to update the scheduler configuration. So, it is compatible with the previous configuration. Here is my thinking. We can consider this in 2 situations. A) The ACL is disabled. (yarn.acl.enable = false). In this kind of configuration, changing the default ACL of ROOT queue will not affect users. Because scheduler do not check any ACL at this time. B) The ACL is enabled. (yarn.acl.enable = true). Currently, if user want to make ACL works correctly. They have to explicitly set the ACL of root queue to NOBODY_ACL( ). After we change the default ACL of root queue, current configuration should work(the root queue ACL should be set as what user wants.). But we can get benefit in new scheduler configuration. In new configuration, user do not need to explicitly set the root queue ACL. What's your opinion? I create 2 subtasks to track each item. Please feel free to comment. I am formatting the patch. Will give you a review soon. Thanks, Yarn scheduler ACL improvement -- Key: YARN-1941 URL: https://issues.apache.org/jira/browse/YARN-1941 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.3.0 Reporter: Gordon Wang Labels: scheduler Defect: 1. Currently, in Yarn Capacity Scheduler and Yarn Fair Scheduler, the queue ACL is always checked when submitting a app to scheduler, regardless of the property yarn.acl.enable. But for killing an app, the ACL is checked when yarn.acl.enable is set. The behaviour is not consistent. 2. default ACL for root queue is EVERYBODY_ACL( * ), while default ACL for other queues is NODODY_ACL( ). From users' view, this is error prone and not easy to understand the ACL policy of Yarn scheduler. root queue should not be so special compared with other parent queues. For example, if I want to set capacity scheduler ACL, the ACL of root has to be set explicitly. Otherwise, everyone can submit APP to yarn scheduler. Because root queue ACL is EVERYBODY_ACL. This is hard for user to administrate yarn scheduler. So, I propose to improve the ACL of yarn scheduler in the following aspects. 1. only enable scheduler queue ACL when yarn.acl.enable is set to true. 2. set the default ACL of root queue as NOBODY_ACL( ). Make all the parent queues' ACL consistent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-1950) Do not check ACL in Yarn scheduler when Yarn ACL is disabled
Gordon Wang created YARN-1950: - Summary: Do not check ACL in Yarn scheduler when Yarn ACL is disabled Key: YARN-1950 URL: https://issues.apache.org/jira/browse/YARN-1950 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.3.0 Reporter: Gordon Wang -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-1951) Change root queue default ACL to NOBODY_ACL in yarn scheduler
Gordon Wang created YARN-1951: - Summary: Change root queue default ACL to NOBODY_ACL in yarn scheduler Key: YARN-1951 URL: https://issues.apache.org/jira/browse/YARN-1951 Project: Hadoop YARN Issue Type: Sub-task Components: scheduler Affects Versions: 2.3.0 Reporter: Gordon Wang -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1950) Do not check ACL in Yarn scheduler when Yarn ACL is disabled
[ https://issues.apache.org/jira/browse/YARN-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13971446#comment-13971446 ] Gordon Wang commented on YARN-1950: --- Use this issue to track the item 1 in YARN-1941 Do not check ACL in Yarn scheduler when Yarn ACL is disabled Key: YARN-1950 URL: https://issues.apache.org/jira/browse/YARN-1950 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.3.0 Reporter: Gordon Wang -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1941) Yarn scheduler ACL improvement
[ https://issues.apache.org/jira/browse/YARN-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13971454#comment-13971454 ] Gordon Wang commented on YARN-1941: --- [~sandyr], would you mind to assign these 3 jiras to me? And I can submit the patch later :) Thanks a lot ! Yarn scheduler ACL improvement -- Key: YARN-1941 URL: https://issues.apache.org/jira/browse/YARN-1941 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.3.0 Reporter: Gordon Wang Labels: scheduler Defect: 1. Currently, in Yarn Capacity Scheduler and Yarn Fair Scheduler, the queue ACL is always checked when submitting a app to scheduler, regardless of the property yarn.acl.enable. But for killing an app, the ACL is checked when yarn.acl.enable is set. The behaviour is not consistent. 2. default ACL for root queue is EVERYBODY_ACL( * ), while default ACL for other queues is NODODY_ACL( ). From users' view, this is error prone and not easy to understand the ACL policy of Yarn scheduler. root queue should not be so special compared with other parent queues. For example, if I want to set capacity scheduler ACL, the ACL of root has to be set explicitly. Otherwise, everyone can submit APP to yarn scheduler. Because root queue ACL is EVERYBODY_ACL. This is hard for user to administrate yarn scheduler. So, I propose to improve the ACL of yarn scheduler in the following aspects. 1. only enable scheduler queue ACL when yarn.acl.enable is set to true. 2. set the default ACL of root queue as NOBODY_ACL( ). Make all the parent queues' ACL consistent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-1941) Yarn scheduler ACL improvement
Gordon Wang created YARN-1941: - Summary: Yarn scheduler ACL improvement Key: YARN-1941 URL: https://issues.apache.org/jira/browse/YARN-1941 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.3.0 Reporter: Gordon Wang Defect: 1. Currently, in Yarn Capacity Scheduler and Yarn Fair Scheduler, the queue ACL is always checked when submitting a app to scheduler, regardless of the property yarn.acl.enable. But for killing an app, the ACL is checked when yarn.acl.enable is set. The behaviour is not consistent. 2. default ACL for root queue is EVERYBODY_ACL(*), while default ACL for other queues is NODODY_ACL( ). From users' view, this is error prone and not easy to understand the ACL policy of Yarn scheduler. root queue should not be so special compared with other parent queues. For example, if I want to set capacity scheduler ACL, the ACL of root has to be set explicitly. Otherwise, everyone can submit APP to yarn scheduler. Because root queue ACL is EVERYBODY_ACL. This is hard for user to administrate yarn scheduler. So, I propose to improve the ACL of yarn scheduler in the following aspects. 1. only enable scheduler queue ACL when yarn.acl.enable is set to true. 2. set the default ACL of root queue as NOBODY_ACL( ). Make all the parent queues' ACL consistent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1941) Yarn scheduler ACL improvement
[ https://issues.apache.org/jira/browse/YARN-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969349#comment-13969349 ] Gordon Wang commented on YARN-1941: --- Checking scheduler queue ACL when submitting a APP to scheduler may derive from hadoop 1 in which there is no properties to enable ACL. But in yarn, since there is already a property to enable yarn acl. we should change the behaviour of the scheduler. Your comments are very welcome. I think I can format a patch later. If I can get any comments from community, I will change my patch according to your opinion. Thanks in advance. Yarn scheduler ACL improvement -- Key: YARN-1941 URL: https://issues.apache.org/jira/browse/YARN-1941 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.3.0 Reporter: Gordon Wang Labels: scheduler Defect: 1. Currently, in Yarn Capacity Scheduler and Yarn Fair Scheduler, the queue ACL is always checked when submitting a app to scheduler, regardless of the property yarn.acl.enable. But for killing an app, the ACL is checked when yarn.acl.enable is set. The behaviour is not consistent. 2. default ACL for root queue is EVERYBODY_ACL(*), while default ACL for other queues is NODODY_ACL( ). From users' view, this is error prone and not easy to understand the ACL policy of Yarn scheduler. root queue should not be so special compared with other parent queues. For example, if I want to set capacity scheduler ACL, the ACL of root has to be set explicitly. Otherwise, everyone can submit APP to yarn scheduler. Because root queue ACL is EVERYBODY_ACL. This is hard for user to administrate yarn scheduler. So, I propose to improve the ACL of yarn scheduler in the following aspects. 1. only enable scheduler queue ACL when yarn.acl.enable is set to true. 2. set the default ACL of root queue as NOBODY_ACL( ). Make all the parent queues' ACL consistent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-1941) Yarn scheduler ACL improvement
[ https://issues.apache.org/jira/browse/YARN-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gordon Wang updated YARN-1941: -- Description: Defect: 1. Currently, in Yarn Capacity Scheduler and Yarn Fair Scheduler, the queue ACL is always checked when submitting a app to scheduler, regardless of the property yarn.acl.enable. But for killing an app, the ACL is checked when yarn.acl.enable is set. The behaviour is not consistent. 2. default ACL for root queue is EVERYBODY_ACL( * ), while default ACL for other queues is NODODY_ACL( ). From users' view, this is error prone and not easy to understand the ACL policy of Yarn scheduler. root queue should not be so special compared with other parent queues. For example, if I want to set capacity scheduler ACL, the ACL of root has to be set explicitly. Otherwise, everyone can submit APP to yarn scheduler. Because root queue ACL is EVERYBODY_ACL. This is hard for user to administrate yarn scheduler. So, I propose to improve the ACL of yarn scheduler in the following aspects. 1. only enable scheduler queue ACL when yarn.acl.enable is set to true. 2. set the default ACL of root queue as NOBODY_ACL( ). Make all the parent queues' ACL consistent. was: Defect: 1. Currently, in Yarn Capacity Scheduler and Yarn Fair Scheduler, the queue ACL is always checked when submitting a app to scheduler, regardless of the property yarn.acl.enable. But for killing an app, the ACL is checked when yarn.acl.enable is set. The behaviour is not consistent. 2. default ACL for root queue is EVERYBODY_ACL(*), while default ACL for other queues is NODODY_ACL( ). From users' view, this is error prone and not easy to understand the ACL policy of Yarn scheduler. root queue should not be so special compared with other parent queues. For example, if I want to set capacity scheduler ACL, the ACL of root has to be set explicitly. Otherwise, everyone can submit APP to yarn scheduler. Because root queue ACL is EVERYBODY_ACL. This is hard for user to administrate yarn scheduler. So, I propose to improve the ACL of yarn scheduler in the following aspects. 1. only enable scheduler queue ACL when yarn.acl.enable is set to true. 2. set the default ACL of root queue as NOBODY_ACL( ). Make all the parent queues' ACL consistent. Yarn scheduler ACL improvement -- Key: YARN-1941 URL: https://issues.apache.org/jira/browse/YARN-1941 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.3.0 Reporter: Gordon Wang Labels: scheduler Defect: 1. Currently, in Yarn Capacity Scheduler and Yarn Fair Scheduler, the queue ACL is always checked when submitting a app to scheduler, regardless of the property yarn.acl.enable. But for killing an app, the ACL is checked when yarn.acl.enable is set. The behaviour is not consistent. 2. default ACL for root queue is EVERYBODY_ACL( * ), while default ACL for other queues is NODODY_ACL( ). From users' view, this is error prone and not easy to understand the ACL policy of Yarn scheduler. root queue should not be so special compared with other parent queues. For example, if I want to set capacity scheduler ACL, the ACL of root has to be set explicitly. Otherwise, everyone can submit APP to yarn scheduler. Because root queue ACL is EVERYBODY_ACL. This is hard for user to administrate yarn scheduler. So, I propose to improve the ACL of yarn scheduler in the following aspects. 1. only enable scheduler queue ACL when yarn.acl.enable is set to true. 2. set the default ACL of root queue as NOBODY_ACL( ). Make all the parent queues' ACL consistent. -- This message was sent by Atlassian JIRA (v6.2#6252)
