[jira] [Commented] (YARN-3843) Fair Scheduler should not accept apps with space keys as queue name
[
https://issues.apache.org/jira/browse/YARN-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598674#comment-14598674
]
zhihai xu commented on YARN-3843:
-
[~dongwook], thanks for the confirmation!
Fair Scheduler should not accept apps with space keys as queue name
---
Key: YARN-3843
URL: https://issues.apache.org/jira/browse/YARN-3843
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.4.0, 2.5.0
Reporter: Dongwook Kwon
Priority: Minor
Fix For: 2.8.0
Attachments: YARN-3843.01.patch
As YARN-461, since empty string queue name is not valid, queue name with
space keys such as ,should not be accepted either, also not as
prefix nor postfix.
e.g) root.test.queuename , or root.test. queuename
I have 2 specific cases kill RM with these space keys as part of queue name.
1) Without placement policy (hadoop 2.4.0 and above),
When a job is submitted with (space key) as queue name
e.g) mapreduce.job.queuename=
2) With placement policy (hadoop 2.5.0 and above)
Once a job is submitted without space key as queue name, and submit another
job with space key.
e.g) 1st time: mapreduce.job.queuename=root.test.user1
2nd time: mapreduce.job.queuename=root.test.user1
{code}
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.974 sec
FAILURE! - in
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
testQueueNameWithSpace(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler)
Time elapsed: 0.724 sec ERROR!
org.apache.hadoop.metrics2.MetricsException: Metrics source
QueueMetrics,q0=root,q1=adhoc,q2=birvine already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:135)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:112)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:218)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueMetrics.forQueue(FSQueueMetrics.java:96)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue.init(FSQueue.java:56)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.init(FSLeafQueue.java:66)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.createQueue(QueueManager.java:169)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.getQueue(QueueManager.java:120)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.getLeafQueue(QueueManager.java:88)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.assignToQueue(FairScheduler.java:660)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.addApplication(FairScheduler.java:569)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1127)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueNameWithSpace(TestFairScheduler.java:627)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3843) Fair Scheduler should not accept apps with space keys as queue name
[
https://issues.apache.org/jira/browse/YARN-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596842#comment-14596842
]
Dongwook Kwon commented on YARN-3843:
-
From my investigation, QueueMetrics doesn't allow space key string as start or
end of names, it just trims empty strings.
static final Splitter Q_SPLITTER =
Splitter.on('.').omitEmptyStrings().trimResults();
https://github.com/apache/hadoop/blob/branch-2.5.2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java#L112
https://github.com/apache/hadoop/blob/branch-2.5.2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java#L85
So, from FairScheduler, root.adhoc.birvine , this queue name with the space
at the end of name, it is treated as different from root.adhoc.birvine
because it has one more character, and from QueueMetrics, because names are
trimmed, all of sudden, 2 different queue names become the same that causes the
error as Metrics source QueueMetrics,q0=root,q1=adhoc,q2=birvine already
exists!
Fair Scheduler should not accept apps with space keys as queue name
---
Key: YARN-3843
URL: https://issues.apache.org/jira/browse/YARN-3843
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.4.0, 2.5.0
Reporter: Dongwook Kwon
Priority: Minor
As YARN-461, since empty string queue name is not valid, queue name with
space keys such as ,should not be accepted either, also not as
prefix nor postfix.
e.g) root.test.queuename , or root.test. queuename
I have 2 specific cases kill RM with these space keys as part of queue name.
1) Without placement policy (hadoop 2.4.0 and above),
When a job is submitted with (space key) as queue name
e.g) mapreduce.job.queuename=
2) With placement policy (hadoop 2.5.0 and above)
Once a job is submitted without space key as queue name, and submit another
job with space key.
e.g) 1st time: mapreduce.job.queuename=root.test.user1
2nd time: mapreduce.job.queuename=root.test.user1
{code}
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.974 sec
FAILURE! - in
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
testQueueNameWithSpace(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler)
Time elapsed: 0.724 sec ERROR!
org.apache.hadoop.metrics2.MetricsException: Metrics source
QueueMetrics,q0=root,q1=adhoc,q2=birvine already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:135)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:112)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:218)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueMetrics.forQueue(FSQueueMetrics.java:96)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue.init(FSQueue.java:56)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.init(FSLeafQueue.java:66)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.createQueue(QueueManager.java:169)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.getQueue(QueueManager.java:120)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.getLeafQueue(QueueManager.java:88)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.assignToQueue(FairScheduler.java:660)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.addApplication(FairScheduler.java:569)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1127)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueNameWithSpace(TestFairScheduler.java:627)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3843) Fair Scheduler should not accept apps with space keys as queue name
[
https://issues.apache.org/jira/browse/YARN-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596858#comment-14596858
]
Dongwook Kwon commented on YARN-3843:
-
Thanks, you're right, it's duplicated. I didn't find the other jira case, I
will close it.
Fair Scheduler should not accept apps with space keys as queue name
---
Key: YARN-3843
URL: https://issues.apache.org/jira/browse/YARN-3843
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.4.0, 2.5.0
Reporter: Dongwook Kwon
Priority: Minor
Attachments: YARN-3843.01.patch
As YARN-461, since empty string queue name is not valid, queue name with
space keys such as ,should not be accepted either, also not as
prefix nor postfix.
e.g) root.test.queuename , or root.test. queuename
I have 2 specific cases kill RM with these space keys as part of queue name.
1) Without placement policy (hadoop 2.4.0 and above),
When a job is submitted with (space key) as queue name
e.g) mapreduce.job.queuename=
2) With placement policy (hadoop 2.5.0 and above)
Once a job is submitted without space key as queue name, and submit another
job with space key.
e.g) 1st time: mapreduce.job.queuename=root.test.user1
2nd time: mapreduce.job.queuename=root.test.user1
{code}
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.974 sec
FAILURE! - in
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
testQueueNameWithSpace(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler)
Time elapsed: 0.724 sec ERROR!
org.apache.hadoop.metrics2.MetricsException: Metrics source
QueueMetrics,q0=root,q1=adhoc,q2=birvine already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:135)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:112)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:218)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueMetrics.forQueue(FSQueueMetrics.java:96)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue.init(FSQueue.java:56)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.init(FSLeafQueue.java:66)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.createQueue(QueueManager.java:169)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.getQueue(QueueManager.java:120)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.getLeafQueue(QueueManager.java:88)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.assignToQueue(FairScheduler.java:660)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.addApplication(FairScheduler.java:569)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1127)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueNameWithSpace(TestFairScheduler.java:627)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3843) Fair Scheduler should not accept apps with space keys as queue name
[
https://issues.apache.org/jira/browse/YARN-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596848#comment-14596848
]
zhihai xu commented on YARN-3843:
-
Hi [~dongwook], thanks for reporting this issue. I think this issue was fixed
at YARN-3241.
Fair Scheduler should not accept apps with space keys as queue name
---
Key: YARN-3843
URL: https://issues.apache.org/jira/browse/YARN-3843
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler
Affects Versions: 2.4.0, 2.5.0
Reporter: Dongwook Kwon
Priority: Minor
As YARN-461, since empty string queue name is not valid, queue name with
space keys such as ,should not be accepted either, also not as
prefix nor postfix.
e.g) root.test.queuename , or root.test. queuename
I have 2 specific cases kill RM with these space keys as part of queue name.
1) Without placement policy (hadoop 2.4.0 and above),
When a job is submitted with (space key) as queue name
e.g) mapreduce.job.queuename=
2) With placement policy (hadoop 2.5.0 and above)
Once a job is submitted without space key as queue name, and submit another
job with space key.
e.g) 1st time: mapreduce.job.queuename=root.test.user1
2nd time: mapreduce.job.queuename=root.test.user1
{code}
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.974 sec
FAILURE! - in
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
testQueueNameWithSpace(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler)
Time elapsed: 0.724 sec ERROR!
org.apache.hadoop.metrics2.MetricsException: Metrics source
QueueMetrics,q0=root,q1=adhoc,q2=birvine already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:135)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:112)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:218)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueMetrics.forQueue(FSQueueMetrics.java:96)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue.init(FSQueue.java:56)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.init(FSLeafQueue.java:66)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.createQueue(QueueManager.java:169)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.getQueue(QueueManager.java:120)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.getLeafQueue(QueueManager.java:88)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.assignToQueue(FairScheduler.java:660)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.addApplication(FairScheduler.java:569)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1127)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueNameWithSpace(TestFairScheduler.java:627)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
