[jira] [Created] (YARN-10128) [FederationSecurity] YARN RMAdmin commands fail when Authorization is enabled on router
Bilwa S T created YARN-10128:
Summary: [FederationSecurity] YARN RMAdmin commands fail when
Authorization is enabled on router
Key: YARN-10128
URL: https://issues.apache.org/jira/browse/YARN-10128
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bilwa S T
Exception thrown is
{quote}Protocol interface
org.apache.hadoop.yarn.server.api.ResourceManagerAdministrationProtocolPB is
not known., while invoking
ResourceManagerAdministrationProtocolPBClientImpl.refreshQueues over rm2 after
1 failover attempts. Trying to failover after sleeping for 44717ms.
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10128) [FederationSecurity] YARN RMAdmin commands fail when Authorization is enabled on router
[
https://issues.apache.org/jira/browse/YARN-10128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bilwa S T reassigned YARN-10128:
Assignee: Bilwa S T
> [FederationSecurity] YARN RMAdmin commands fail when Authorization is enabled
> on router
> ---
>
> Key: YARN-10128
> URL: https://issues.apache.org/jira/browse/YARN-10128
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
>
> Exception thrown is
> {quote}Protocol interface
> org.apache.hadoop.yarn.server.api.ResourceManagerAdministrationProtocolPB is
> not known., while invoking
> ResourceManagerAdministrationProtocolPBClientImpl.refreshQueues over rm2
> after 1 failover attempts. Trying to failover after sleeping for 44717ms.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9521) RM failed to start due to system services
[
https://issues.apache.org/jira/browse/YARN-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035048#comment-17035048
]
Hadoop QA commented on YARN-9521:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 14s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
52s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
56s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | YARN-9521 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12993215/YARN-9521.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 55c41714be93 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9b8a78d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (YARN-9521) RM failed to start due to system services
[
https://issues.apache.org/jira/browse/YARN-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035021#comment-17035021
]
kyungwan nam commented on YARN-9521:
Attaches a new patch including test code.
[~eyang], [~prabhujoseph] Could you take a look it when you are available?
Thanks!
> RM failed to start due to system services
> -
>
> Key: YARN-9521
> URL: https://issues.apache.org/jira/browse/YARN-9521
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.2
>Reporter: kyungwan nam
>Assignee: kyungwan nam
>Priority: Major
> Attachments: YARN-9521.001.patch, YARN-9521.002.patch,
> YARN-9521.003.patch
>
>
> when starting RM, listing system services directory has failed as follows.
> {code}
> 2019-04-30 17:18:25,441 INFO client.SystemServiceManagerImpl
> (SystemServiceManagerImpl.java:serviceInit(114)) - System Service Directory
> is configured to /services
> 2019-04-30 17:18:25,467 INFO client.SystemServiceManagerImpl
> (SystemServiceManagerImpl.java:serviceInit(120)) - UserGroupInformation
> initialized to yarn (auth:SIMPLE)
> 2019-04-30 17:18:25,467 INFO service.AbstractService
> (AbstractService.java:noteFailure(267)) - Service ResourceManager failed in
> state STARTED
> org.apache.hadoop.service.ServiceStateException: java.io.IOException:
> Filesystem closed
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:203)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:869)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1228)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1269)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1265)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1265)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1316)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1501)
> Caused by: java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:473)
> at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1639)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1217)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1233)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1200)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1179)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1175)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusIterator(DistributedFileSystem.java:1187)
> at
> org.apache.hadoop.yarn.service.client.SystemServiceManagerImpl.list(SystemServiceManagerImpl.java:375)
> at
> org.apache.hadoop.yarn.service.client.SystemServiceManagerImpl.scanForUserServices(SystemServiceManagerImpl.java:282)
> at
> org.apache.hadoop.yarn.service.client.SystemServiceManagerImpl.serviceStart(SystemServiceManagerImpl.java:126)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> ... 13 more
> {code}
> it looks like due to the usage of filesystem cache.
> this issue does not happen, when I add "fs.hdfs.impl.disable.cache=true" to
> yarn-site
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9521) RM failed to start due to system services
[
https://issues.apache.org/jira/browse/YARN-9521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kyungwan nam updated YARN-9521:
---
Attachment: YARN-9521.003.patch
> RM failed to start due to system services
> -
>
> Key: YARN-9521
> URL: https://issues.apache.org/jira/browse/YARN-9521
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.2
>Reporter: kyungwan nam
>Assignee: kyungwan nam
>Priority: Major
> Attachments: YARN-9521.001.patch, YARN-9521.002.patch,
> YARN-9521.003.patch
>
>
> when starting RM, listing system services directory has failed as follows.
> {code}
> 2019-04-30 17:18:25,441 INFO client.SystemServiceManagerImpl
> (SystemServiceManagerImpl.java:serviceInit(114)) - System Service Directory
> is configured to /services
> 2019-04-30 17:18:25,467 INFO client.SystemServiceManagerImpl
> (SystemServiceManagerImpl.java:serviceInit(120)) - UserGroupInformation
> initialized to yarn (auth:SIMPLE)
> 2019-04-30 17:18:25,467 INFO service.AbstractService
> (AbstractService.java:noteFailure(267)) - Service ResourceManager failed in
> state STARTED
> org.apache.hadoop.service.ServiceStateException: java.io.IOException:
> Filesystem closed
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:203)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:869)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1228)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1269)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1265)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1265)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1316)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1501)
> Caused by: java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:473)
> at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1639)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1217)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1233)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1200)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1179)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1175)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusIterator(DistributedFileSystem.java:1187)
> at
> org.apache.hadoop.yarn.service.client.SystemServiceManagerImpl.list(SystemServiceManagerImpl.java:375)
> at
> org.apache.hadoop.yarn.service.client.SystemServiceManagerImpl.scanForUserServices(SystemServiceManagerImpl.java:282)
> at
> org.apache.hadoop.yarn.service.client.SystemServiceManagerImpl.serviceStart(SystemServiceManagerImpl.java:126)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> ... 13 more
> {code}
> it looks like due to the usage of filesystem cache.
> this issue does not happen, when I add "fs.hdfs.impl.disable.cache=true" to
> yarn-site
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6492) Generate queue metrics for each partition
[ https://issues.apache.org/jira/browse/YARN-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034902#comment-17034902 ] Aihua Xu commented on YARN-6492: [~maniraj...@gmail.com] Do you have update on this jira? > Generate queue metrics for each partition > - > > Key: YARN-6492 > URL: https://issues.apache.org/jira/browse/YARN-6492 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Jonathan Hung >Assignee: Manikandan R >Priority: Major > Attachments: PartitionQueueMetrics_default_partition.txt, > PartitionQueueMetrics_x_partition.txt, PartitionQueueMetrics_y_partition.txt, > YARN-6492.001.patch, YARN-6492.002.patch, YARN-6492.003.patch, > YARN-6492.004.patch, YARN-6492.005.WIP.patch, YARN-6492.006.WIP.patch, > YARN-6492.007.WIP.patch, partition_metrics.txt > > > We are interested in having queue metrics for all partitions. Right now each > queue has one QueueMetrics object which captures metrics either in default > partition or across all partitions. (After YARN-6467 it will be in default > partition) > But having the partition metrics would be very useful. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-10022: - Fix Version/s: 3.1.4 > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Fix For: 3.3.0, 3.2.2, 3.1.4 > > Attachments: YARN-10022-branch-3.1.001.patch, > YARN-10022-branch-3.2.001.patch, YARN-10022.001.patch, YARN-10022.002.patch, > YARN-10022.003.patch, YARN-10022.004.patch, YARN-10022.005.patch, > YARN-10022.006.patch, YARN-10022.007.patch, YARN-10022.WIP.patch, > YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034599#comment-17034599 ] Prabhu Joseph commented on YARN-10022: -- Thanks [~kmarton] for the patch. Have backported to branch-3.1. Will resolve the Jira. > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-10022-branch-3.1.001.patch, > YARN-10022-branch-3.2.001.patch, YARN-10022.001.patch, YARN-10022.002.patch, > YARN-10022.003.patch, YARN-10022.004.patch, YARN-10022.005.patch, > YARN-10022.006.patch, YARN-10022.007.patch, YARN-10022.WIP.patch, > YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[
https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034597#comment-17034597
]
Hudson commented on YARN-10127:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17941 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17941/])
YARN-10127. Remove setting App Ordering Policy to ParentQueue in (pjoseph: rev
e6377972113db844d7e7b99867fad19d3cbc58bd)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java
> FSQueueConverter should not set App Ordering Policy to Parent Queue
> ---
>
> Key: YARN-10127
> URL: https://issues.apache.org/jira/browse/YARN-10127
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Peter Bacsko
>Priority: Major
> Labels: Reviewed
> Fix For: 3.3.0
>
> Attachments: YARN-10127-001.patch
>
>
> FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent
> Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
> {code}
> Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct
> queue ordering policy=fair queue=root
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
>
> {code}
> Input fair-scheduler.xml:
> {code}
> [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml
>
>
>
> fair
>
> fair
>
>
> fair
>
>
>
>
>
>
>
>
>
>
> {code}
> Command Used:
> {code}
> yarn fs2cs -t -f /tmp/fair-scheduler.xml -y
> /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o
> /tmp/CS
> {code}
> Output capacity-scheduler.xml
> {code}
>
> yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically
> yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically
> yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically
>
[jira] [Updated] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[
https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-10127:
-
Labels: Reviewed (was: )
> FSQueueConverter should not set App Ordering Policy to Parent Queue
> ---
>
> Key: YARN-10127
> URL: https://issues.apache.org/jira/browse/YARN-10127
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Peter Bacsko
>Priority: Major
> Labels: Reviewed
> Fix For: 3.3.0
>
> Attachments: YARN-10127-001.patch
>
>
> FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent
> Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
> {code}
> Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct
> queue ordering policy=fair queue=root
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
>
> {code}
> Input fair-scheduler.xml:
> {code}
> [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml
>
>
>
> fair
>
> fair
>
>
> fair
>
>
>
>
>
>
>
>
>
>
> {code}
> Command Used:
> {code}
> yarn fs2cs -t -f /tmp/fair-scheduler.xml -y
> /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o
> /tmp/CS
> {code}
> Output capacity-scheduler.xml
> {code}
>
> yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically
> yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically
> yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically
> yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically
> yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.root.ordering-policyfairfalseprogrammatically
>
> {code}
> Root Queue is set with App Ordering Policy fair which is wrong
> {code}
>
[jira] [Updated] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[
https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-10127:
-
Fix Version/s: 3.3.0
> FSQueueConverter should not set App Ordering Policy to Parent Queue
> ---
>
> Key: YARN-10127
> URL: https://issues.apache.org/jira/browse/YARN-10127
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-10127-001.patch
>
>
> FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent
> Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
> {code}
> Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct
> queue ordering policy=fair queue=root
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
>
> {code}
> Input fair-scheduler.xml:
> {code}
> [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml
>
>
>
> fair
>
> fair
>
>
> fair
>
>
>
>
>
>
>
>
>
>
> {code}
> Command Used:
> {code}
> yarn fs2cs -t -f /tmp/fair-scheduler.xml -y
> /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o
> /tmp/CS
> {code}
> Output capacity-scheduler.xml
> {code}
>
> yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically
> yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically
> yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically
> yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically
> yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.root.ordering-policyfairfalseprogrammatically
>
> {code}
> Root Queue is set with App Ordering Policy fair which is wrong
> {code}
>
[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[
https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034589#comment-17034589
]
Prabhu Joseph commented on YARN-10127:
--
Have committed the patch to trunk. Will resolve the Jira.
> FSQueueConverter should not set App Ordering Policy to Parent Queue
> ---
>
> Key: YARN-10127
> URL: https://issues.apache.org/jira/browse/YARN-10127
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10127-001.patch
>
>
> FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent
> Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
> {code}
> Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct
> queue ordering policy=fair queue=root
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
>
> {code}
> Input fair-scheduler.xml:
> {code}
> [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml
>
>
>
> fair
>
> fair
>
>
> fair
>
>
>
>
>
>
>
>
>
>
> {code}
> Command Used:
> {code}
> yarn fs2cs -t -f /tmp/fair-scheduler.xml -y
> /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o
> /tmp/CS
> {code}
> Output capacity-scheduler.xml
> {code}
>
> yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically
> yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically
> yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically
> yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically
> yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.root.ordering-policyfairfalseprogrammatically
>
> {code}
> Root Queue is set with App Ordering Policy fair which is wrong
>
[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[
https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034584#comment-17034584
]
Hadoop QA commented on YARN-10127:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 87m
56s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | YARN-10127 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12993136/YARN-10127-001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8a68761a3ae8 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / cc8ae59 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25514/testReport/ |
| Max. process+thread count | 820 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25514/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> FSQueueConverter should not set App Ordering
[jira] [Commented] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[
https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034580#comment-17034580
]
Hadoop QA commented on YARN-10022:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 26m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
3s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} branch-3.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 113 unchanged - 9 fixed = 114 total (was 122)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}305m 34s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}390m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures |
| | hadoop.yarn.server.resourcemanager.TestApplicationACLs |
| | hadoop.yarn.server.resourcemanager.TestWorkPreservingUnmanagedAM |
| | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector |
| | hadoop.yarn.server.resourcemanager.placement.TestPlacementManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:70a0ef5d4a6 |
| JIRA Issue | YARN-10022 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12992247/YARN-10022-branch-3.1.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7f101fa35021 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.1 / 547f179 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
[jira] [Commented] (YARN-10029) Add option to UIv2 to get container logs from the new JHS API
[ https://issues.apache.org/jira/browse/YARN-10029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034554#comment-17034554 ] Gergely Pollak commented on YARN-10029: --- [~adam.antal], thank you for the patch, LGTM+1 (Non-binding). There are a few more places where the code could be cleaned up or methods could be created for repeated code blocks, but those are not the result of your changes, so I don't think those should be fixed along this patch, perhaps filing a followup Jira might help to fix those at a later point. > Add option to UIv2 to get container logs from the new JHS API > - > > Key: YARN-10029 > URL: https://issues.apache.org/jira/browse/YARN-10029 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.2.1 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Attachments: YARN-10029.001.patch, YARN-10029.002.patch, > YARN-10029.003.patch > > > Provided the new API is ready to use (also integrated into JHS in > YARN-10028), we can add a new config option to UIv2 that would make the UIv2 > to request logs from the JHS API similarly as the ATSv2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10113) SystemServiceManagerImpl fails to initialize
[
https://issues.apache.org/jira/browse/YARN-10113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034548#comment-17034548
]
Eric Yang commented on YARN-10113:
--
[~kyungwan nam] Thank you for the patch. Patch looks good. Can we have a test
case to covert this?
> SystemServiceManagerImpl fails to initialize
> -
>
> Key: YARN-10113
> URL: https://issues.apache.org/jira/browse/YARN-10113
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-10113-001.patch, YARN-10113-002.patch
>
>
> RM fails to start with SystemServiceManagerImpl failed to initialize.
> {code}
> 2020-01-28 12:20:43,631 WARN ha.ActiveStandbyElector
> (ActiveStandbyElector.java:becomeActive(900)) - Exception handling the
> winning of election
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:146)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:476)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:636)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error when
> transitioning to Active mode
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:325)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> ... 4 more
> Caused by: org.apache.hadoop.service.ServiceStateException:
> java.io.IOException: Filesystem closed
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:203)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:881)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1257)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1298)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1294)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1294)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:320)
> ... 5 more
> Caused by: java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:475)
> at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1645)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1219)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1235)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.(DistributedFileSystem.java:1202)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1181)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1177)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusIterator(DistributedFileSystem.java:1189)
> at
> org.apache.hadoop.yarn.service.client.SystemServiceManagerImpl.list(SystemServiceManagerImpl.java:375)
> at
> org.apache.hadoop.yarn.service.client.SystemServiceManagerImpl.scanForUserServices(SystemServiceManagerImpl.java:282)
> at
> org.apache.hadoop.yarn.service.client.SystemServiceManagerImpl.serviceStart(SystemServiceManagerImpl.java:126)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> ... 16 more
>
[jira] [Commented] (YARN-10080) Support show app id on localizer thread pool
[ https://issues.apache.org/jira/browse/YARN-10080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034544#comment-17034544 ] Adam Antal commented on YARN-10080: --- I think this is a useful patch for debugging. +1 (non-binding). > Support show app id on localizer thread pool > > > Key: YARN-10080 > URL: https://issues.apache.org/jira/browse/YARN-10080 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: zhoukang >Assignee: zhoukang >Priority: Major > Attachments: YARN-10080-001.patch, YARN-10080.002.patch > > > Currently when we are troubleshooting a container localizer issue, if we want > to analyze the jstack with thread detail, we can not figure out which thread > is processing the given container. So i want to add app id on the thread name -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10101) Support listing of aggregated logs for containers belonging to an application attempt
[ https://issues.apache.org/jira/browse/YARN-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034542#comment-17034542 ] Adam Antal commented on YARN-10101: --- Thanks [~snemeth]! > Support listing of aggregated logs for containers belonging to an application > attempt > - > > Key: YARN-10101 > URL: https://issues.apache.org/jira/browse/YARN-10101 > Project: Hadoop YARN > Issue Type: Sub-task > Components: log-aggregation, yarn >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-10101.001.patch, YARN-10101.002.patch, > YARN-10101.003.patch, YARN-10101.004.patch, YARN-10101.005.patch, > YARN-10101.006.patch, YARN-10101.007.patch, YARN-10101.008.patch, > YARN-10101.009.patch, YARN-10101.branch-3.2.001.patch, > YARN-10101.branch-3.2.001.patch, YARN-10101.branch-3.2.002.patch > > > To display logs without access to the timeline server, we need an interface > where we can query the list of containers with aggregated logs belonging to > an application attempt. > We should add support for this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[
https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034529#comment-17034529
]
Szilard Nemeth commented on YARN-10127:
---
Thanks [~prabhujoseph] for your explanation.
Thanks [~pbacsko], patch looks good to me as well.
> FSQueueConverter should not set App Ordering Policy to Parent Queue
> ---
>
> Key: YARN-10127
> URL: https://issues.apache.org/jira/browse/YARN-10127
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10127-001.patch
>
>
> FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent
> Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
> {code}
> Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct
> queue ordering policy=fair queue=root
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
>
> {code}
> Input fair-scheduler.xml:
> {code}
> [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml
>
>
>
> fair
>
> fair
>
>
> fair
>
>
>
>
>
>
>
>
>
>
> {code}
> Command Used:
> {code}
> yarn fs2cs -t -f /tmp/fair-scheduler.xml -y
> /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o
> /tmp/CS
> {code}
> Output capacity-scheduler.xml
> {code}
>
> yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically
> yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically
> yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically
> yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically
> yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.root.ordering-policyfairfalseprogrammatically
>
> {code}
> Root Queue is set with App
[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[
https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034473#comment-17034473
]
Prabhu Joseph commented on YARN-10127:
--
Thanks [~pbacsko] for the patch.
+1, waiting for jenkins.
> FSQueueConverter should not set App Ordering Policy to Parent Queue
> ---
>
> Key: YARN-10127
> URL: https://issues.apache.org/jira/browse/YARN-10127
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10127-001.patch
>
>
> FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent
> Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
> {code}
> Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct
> queue ordering policy=fair queue=root
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
>
> {code}
> Input fair-scheduler.xml:
> {code}
> [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml
>
>
>
> fair
>
> fair
>
>
> fair
>
>
>
>
>
>
>
>
>
>
> {code}
> Command Used:
> {code}
> yarn fs2cs -t -f /tmp/fair-scheduler.xml -y
> /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o
> /tmp/CS
> {code}
> Output capacity-scheduler.xml
> {code}
>
> yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically
> yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically
> yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically
> yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically
> yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.root.ordering-policyfairfalseprogrammatically
>
> {code}
> Root Queue is set with App Ordering Policy fair which is wrong
>
[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[
https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034467#comment-17034467
]
Peter Bacsko commented on YARN-10127:
-
Thanks for the explanation [~prabhujoseph] - looks like the solution is to only
emit this property for leaf queues. Otherwise skip it.
Uploaded patch v1.
> FSQueueConverter should not set App Ordering Policy to Parent Queue
> ---
>
> Key: YARN-10127
> URL: https://issues.apache.org/jira/browse/YARN-10127
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10127-001.patch
>
>
> FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent
> Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
> {code}
> Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct
> queue ordering policy=fair queue=root
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
>
> {code}
> Input fair-scheduler.xml:
> {code}
> [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml
>
>
>
> fair
>
> fair
>
>
> fair
>
>
>
>
>
>
>
>
>
>
> {code}
> Command Used:
> {code}
> yarn fs2cs -t -f /tmp/fair-scheduler.xml -y
> /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o
> /tmp/CS
> {code}
> Output capacity-scheduler.xml
> {code}
>
> yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically
> yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically
> yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically
> yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically
> yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically
>
[jira] [Updated] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[
https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10127:
Attachment: YARN-10127-001.patch
> FSQueueConverter should not set App Ordering Policy to Parent Queue
> ---
>
> Key: YARN-10127
> URL: https://issues.apache.org/jira/browse/YARN-10127
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10127-001.patch
>
>
> FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent
> Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
> {code}
> Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct
> queue ordering policy=fair queue=root
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
>
> {code}
> Input fair-scheduler.xml:
> {code}
> [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml
>
>
>
> fair
>
> fair
>
>
> fair
>
>
>
>
>
>
>
>
>
>
> {code}
> Command Used:
> {code}
> yarn fs2cs -t -f /tmp/fair-scheduler.xml -y
> /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o
> /tmp/CS
> {code}
> Output capacity-scheduler.xml
> {code}
>
> yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically
> yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically
> yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically
> yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically
> yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.root.ordering-policyfairfalseprogrammatically
>
> {code}
> Root Queue is set with App Ordering Policy fair which is wrong
> {code}
> yarn.scheduler.capacity.root.ordering-policyfair
> {code}
[jira] [Comment Edited] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[
https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034447#comment-17034447
]
Prabhu Joseph edited comment on YARN-10127 at 2/11/20 1:44 PM:
---
[~snemeth] CS iterates ParentQueue (root queue as well) with Queue Ordering
Policy. (supported values: utilization, priority-utilization) and iterates
LeafQueue with App Ordering Policy (supported values: fair, fifo). But the
config name is same for both Parent and Leaf queue -
yarn.scheduler.capacity..ordering-policy causing the confusion.
Setting fair to a ParentQueue is wrong as it accepts only utilization,
priority-utilization.
was (Author: prabhu joseph):
[~snemeth] CS iterates ParentQueue (root queue as well) with Queue Ordering
Policy. (supported values: utilization, priority-utilization) and iterates
LeafQueue with App Ordering Policy (supported values: fair, fifo). But the
config name is same for both Parent and Leaf queue -
yarn.scheduler.capacity..ordering-policy causing the confusion.
> FSQueueConverter should not set App Ordering Policy to Parent Queue
> ---
>
> Key: YARN-10127
> URL: https://issues.apache.org/jira/browse/YARN-10127
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Peter Bacsko
>Priority: Major
>
> FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent
> Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
> {code}
> Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct
> queue ordering policy=fair queue=root
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
>
> {code}
> Input fair-scheduler.xml:
> {code}
> [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml
>
>
>
> fair
>
> fair
>
>
> fair
>
>
>
>
>
>
>
>
>
>
> {code}
> Command Used:
> {code}
> yarn fs2cs -t -f /tmp/fair-scheduler.xml -y
> /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o
> /tmp/CS
> {code}
> Output capacity-scheduler.xml
> {code}
>
> yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically
> yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically
>
[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[
https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034447#comment-17034447
]
Prabhu Joseph commented on YARN-10127:
--
[~snemeth] CS iterates ParentQueue (root queue as well) with Queue Ordering
Policy. (supported values: utilization, priority-utilization) and iterates
LeafQueue with App Ordering Policy (supported values: fair, fifo). But the
config name is same for both Parent and Leaf queue -
yarn.scheduler.capacity..ordering-policy causing the confusion.
> FSQueueConverter should not set App Ordering Policy to Parent Queue
> ---
>
> Key: YARN-10127
> URL: https://issues.apache.org/jira/browse/YARN-10127
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Peter Bacsko
>Priority: Major
>
> FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent
> Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
> {code}
> Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct
> queue ordering policy=fair queue=root
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
>
> {code}
> Input fair-scheduler.xml:
> {code}
> [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml
>
>
>
> fair
>
> fair
>
>
> fair
>
>
>
>
>
>
>
>
>
>
> {code}
> Command Used:
> {code}
> yarn fs2cs -t -f /tmp/fair-scheduler.xml -y
> /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o
> /tmp/CS
> {code}
> Output capacity-scheduler.xml
> {code}
>
> yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically
> yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically
> yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically
> yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically
>
[jira] [Assigned] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[
https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko reassigned YARN-10127:
---
Assignee: Peter Bacsko (was: Prabhu Joseph)
> FSQueueConverter should not set App Ordering Policy to Parent Queue
> ---
>
> Key: YARN-10127
> URL: https://issues.apache.org/jira/browse/YARN-10127
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Peter Bacsko
>Priority: Major
>
> FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent
> Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
> {code}
> Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct
> queue ordering policy=fair queue=root
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
>
> {code}
> Input fair-scheduler.xml:
> {code}
> [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml
>
>
>
> fair
>
> fair
>
>
> fair
>
>
>
>
>
>
>
>
>
>
> {code}
> Command Used:
> {code}
> yarn fs2cs -t -f /tmp/fair-scheduler.xml -y
> /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o
> /tmp/CS
> {code}
> Output capacity-scheduler.xml
> {code}
>
> yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically
> yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically
> yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically
> yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically
> yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.root.ordering-policyfairfalseprogrammatically
>
> {code}
> Root Queue is set with App Ordering Policy fair which is wrong
> {code}
> yarn.scheduler.capacity.root.ordering-policyfair
> {code}
--
This message was
[jira] [Assigned] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[
https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph reassigned YARN-10127:
Assignee: Prabhu Joseph
> FSQueueConverter should not set App Ordering Policy to Parent Queue
> ---
>
> Key: YARN-10127
> URL: https://issues.apache.org/jira/browse/YARN-10127
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>
> FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent
> Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
> {code}
> Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct
> queue ordering policy=fair queue=root
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
>
> {code}
> Input fair-scheduler.xml:
> {code}
> [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml
>
>
>
> fair
>
> fair
>
>
> fair
>
>
>
>
>
>
>
>
>
>
> {code}
> Command Used:
> {code}
> yarn fs2cs -t -f /tmp/fair-scheduler.xml -y
> /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o
> /tmp/CS
> {code}
> Output capacity-scheduler.xml
> {code}
>
> yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically
> yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically
> yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically
> yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically
> yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.root.ordering-policyfairfalseprogrammatically
>
> {code}
> Root Queue is set with App Ordering Policy fair which is wrong
> {code}
> yarn.scheduler.capacity.root.ordering-policyfair
> {code}
--
This message was sent by Atlassian
[jira] [Commented] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[
https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034388#comment-17034388
]
Szilard Nemeth commented on YARN-10127:
---
Hi [~prabhujoseph],
What's the rationale behind
{quote}
Root Queue is set with App Ordering Policy fair which is wrong
{quote} ?
Why CS doesn't accept this config?
> FSQueueConverter should not set App Ordering Policy to Parent Queue
> ---
>
> Key: YARN-10127
> URL: https://issues.apache.org/jira/browse/YARN-10127
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Priority: Major
>
> FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent
> Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
> {code}
> Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct
> queue ordering policy=fair queue=root
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
>
> {code}
> Input fair-scheduler.xml:
> {code}
> [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml
>
>
>
> fair
>
> fair
>
>
> fair
>
>
>
>
>
>
>
>
>
>
> {code}
> Command Used:
> {code}
> yarn fs2cs -t -f /tmp/fair-scheduler.xml -y
> /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o
> /tmp/CS
> {code}
> Output capacity-scheduler.xml
> {code}
>
> yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically
> yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically
> yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically
> yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically
> yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically
> yarn.scheduler.capacity.root.ordering-policyfairfalseprogrammatically
>
> {code}
> Root Queue is set with App Ordering
[jira] [Comment Edited] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
[
https://issues.apache.org/jira/browse/YARN-10127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034388#comment-17034388
]
Szilard Nemeth edited comment on YARN-10127 at 2/11/20 12:09 PM:
-
Hi [~prabhujoseph],
What's the rationale behind
{quote}
Root Queue is set with App Ordering Policy fair which is wrong
{quote} ?
Why CS doesn't accept this config?
was (Author: snemeth):
Hi [~prabhujoseph],
What's the rationale behind
{quote}
Root Queue is set with App Ordering Policy fair which is wrong
{quote} ?
Why CS doesn't accept this config?
> FSQueueConverter should not set App Ordering Policy to Parent Queue
> ---
>
> Key: YARN-10127
> URL: https://issues.apache.org/jira/browse/YARN-10127
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Priority: Major
>
> FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent
> Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
> {code}
> Error starting ResourceManager
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct
> queue ordering policy=fair queue=root
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
>
> {code}
> Input fair-scheduler.xml:
> {code}
> [yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml
>
>
>
> fair
>
> fair
>
>
> fair
>
>
>
>
>
>
>
>
>
>
> {code}
> Command Used:
> {code}
> yarn fs2cs -t -f /tmp/fair-scheduler.xml -y
> /var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o
> /tmp/CS
> {code}
> Output capacity-scheduler.xml
> {code}
>
> yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically
> yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically
> yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically
> yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically
> yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically
> yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically
>
[jira] [Updated] (YARN-9696) unused import in Configuration class
[ https://issues.apache.org/jira/browse/YARN-9696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-9696: - Summary: unused import in Configuration class (was: one more import in org.apache.hadoop.conf.Configuration class) > unused import in Configuration class > > > Key: YARN-9696 > URL: https://issues.apache.org/jira/browse/YARN-9696 > Project: Hadoop YARN > Issue Type: Bug >Reporter: runzhou wu >Assignee: Jan Hentschel >Priority: Trivial > > LinkedList is not used . > it is in line 54. the content is "import java.util.LinkedList; " .i think it > can be delete. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10043) FairOrderingPolicy Improvements
[ https://issues.apache.org/jira/browse/YARN-10043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034339#comment-17034339 ] Peter Bacsko commented on YARN-10043: - +1 (non-binding). [~snemeth] could you check this? > FairOrderingPolicy Improvements > --- > > Key: YARN-10043 > URL: https://issues.apache.org/jira/browse/YARN-10043 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Attachments: YARN-10043.001.patch, YARN-10043.002.patch, > YARN-10043.003.patch > > > FairOrderingPolicy can be improved by using some of the approaches (only > relevant) implemented in FairSharePolicy of FS. This improvement has > significance in FS to CS migration context. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[ https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034337#comment-17034337 ] Prabhu Joseph commented on YARN-10124: -- Interesting, this will be helpful. There is a difference in usability between both cases A) STOP a queue and then utilize the capacity of STOPPED queue for sibling queues and B) set parent queue capacity to 0 and utilize the capacity for sibling queues. In case A, user has to wait for queue to be STOPPED completely from DRAINING state and then increase sibling queue capacities. Setting capacity to 0 when queue is in DRAINING state can cause preemption of containers of running apps. In case B, user need not wait, both set parent queue capacity to 0 and update sibling queue capacities can be done at a time. Would like to have both cases. If you are fine with above, will handle case B as part of this Jira and create another Jira for case A. > Remove restriction of ParentQueue capacity zero when childCapacities > 0 > > > Key: YARN-10124 > URL: https://issues.apache.org/jira/browse/YARN-10124 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-10124-001.patch > > > ParentQueue capacity cannot be set to 0 when child capacities > 0. To disable > a parent queue temporarily, user can only STOP the queue but the capacity of > the queue cannot be used for other queues. Allowing 0 capacity for parent > queue will allow user to use the capacity for other queues and also to retain > the child queue capacity values. (else user has to set all child queue > capacities to 0) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10122) In Federation,executing yarn container signal command throws an exception
[
https://issues.apache.org/jira/browse/YARN-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bilwa S T reassigned YARN-10122:
Assignee: Bilwa S T
> In Federation,executing yarn container signal command throws an exception
> -
>
> Key: YARN-10122
> URL: https://issues.apache.org/jira/browse/YARN-10122
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation, yarn
>Reporter: Sushanta Sen
>Assignee: Bilwa S T
>Priority: Major
>
> Executing yarn container signal command failed, prompting an error
> “org.apache.commons.lang.NotImplementedException: Code is not implemented”.
> {noformat}
> ./yarn container -signal container_e79_1581316978887_0001_01_10
> Signalling container container_e79_1581316978887_0001_01_10
> 2020-02-10 14:51:18,045 INFO impl.YarnClientImpl: Signalling container
> container_e79_1581316978887_0001_01_10 with command OUTPUT_THREAD_DUMP
> Exception in thread "main" org.apache.commons.lang.NotImplementedException:
> Code is not implemented
> at
> org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor.signalToContainer(FederationClientInterceptor.java:993)
> at
> org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService.signalToContainer(RouterClientRMService.java:403)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.signalToContainer(ApplicationClientProtocolPBServiceImpl.java:629)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:629)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:928)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:863)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2793)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.signalToContainer(ApplicationClientProtocolPBClientImpl.java:620)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy8.signalToContainer(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.signalToContainer(YarnClientImpl.java:949)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.signalToContainer(ApplicationCLI.java:717)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:478)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:119)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
[jira] [Commented] (YARN-10124) Remove restriction of ParentQueue capacity zero when childCapacities > 0
[ https://issues.apache.org/jira/browse/YARN-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034266#comment-17034266 ] Wilfred Spiegelenburg commented on YARN-10124: -- Based on what I read here is that you expect a stopped queue to not have a capacity and thus return 0 when calculating the correct distributing. If that is the use case then why can't we implement that. In other words: why not turn this around, only return or take into account the capacity of a queue when it is not in a stopped state? So you return 0 for all stopped queues. You do not have to go further than that. No need to (re)calculate below the parent that is stopped 0 (as that is all ignored) and turning the queue back on will trigger the existing settings to be applied again without further changes. > Remove restriction of ParentQueue capacity zero when childCapacities > 0 > > > Key: YARN-10124 > URL: https://issues.apache.org/jira/browse/YARN-10124 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-10124-001.patch > > > ParentQueue capacity cannot be set to 0 when child capacities > 0. To disable > a parent queue temporarily, user can only STOP the queue but the capacity of > the queue cannot be used for other queues. Allowing 0 capacity for parent > queue will allow user to use the capacity for other queues and also to retain > the child queue capacity values. (else user has to set all child queue > capacities to 0) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10127) FSQueueConverter should not set App Ordering Policy to Parent Queue
Prabhu Joseph created YARN-10127:
Summary: FSQueueConverter should not set App Ordering Policy to
Parent Queue
Key: YARN-10127
URL: https://issues.apache.org/jira/browse/YARN-10127
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 3.3.0
Reporter: Prabhu Joseph
FSQueueConverter should not set App Ordering Policy (fair, fifo) to Parent
Queue. RM will fail to start if Parent Queue is set with App Ordering Policy.
{code}
Error starting ResourceManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct
queue ordering policy=fair queue=root
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:1584)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:145)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.(ParentQueue.java:112)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractManagedParentQueue.(AbstractManagedParentQueue.java:51)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ManagedParentQueue.(ManagedParentQueue.java:56)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:272)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:158)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:751)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
{code}
Input fair-scheduler.xml:
{code}
[yarn@mradha-s1-1 /]$ cat /tmp/fair-scheduler.xml
fair
fair
fair
{code}
Command Used:
{code}
yarn fs2cs -t -f /tmp/fair-scheduler.xml -y
/var/run/cloudera-scm-agent/process/11-yarn-RESOURCEMANAGER/yarn-site.xml -o
/tmp/CS
{code}
Output capacity-scheduler.xml
{code}
yarn.scheduler.capacity.root.auto-create-child-queue.enabledtruefalseprogrammatically
yarn.scheduler.capacity.root.users.capacity50.000falseprogrammatically
yarn.scheduler.capacity.root.queuesdefault,usersfalseprogrammatically
yarn.scheduler.capacity.queue-mappings-override.enablefalsefalseprogrammatically
yarn.scheduler.capacity.root.default.capacity50.000falseprogrammatically
yarn.scheduler.capacity.root.default.auto-create-child-queue.enabledtruefalseprogrammatically
yarn.scheduler.capacity.maximum-am-resource-percent0.5falseprogrammatically
yarn.scheduler.capacity.root.users.auto-create-child-queue.enabledtruefalseprogrammatically
yarn.scheduler.capacity.root.default.ordering-policyfairfalseprogrammatically
yarn.scheduler.capacity.queue-mappingsu:%user:%user;u:%user:root.users.%user;u:%user:root.defaultfalseprogrammatically
yarn.scheduler.capacity.root.users.ordering-policyfairfalseprogrammatically
yarn.scheduler.capacity.root.ordering-policyfairfalseprogrammatically
{code}
Root Queue is set with App Ordering Policy fair which is wrong
{code}
yarn.scheduler.capacity.root.ordering-policyfair
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10101) Support listing of aggregated logs for containers belonging to an application attempt
[ https://issues.apache.org/jira/browse/YARN-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034240#comment-17034240 ] Szilard Nemeth commented on YARN-10101: --- Thanks [~adam.antal] for looking into the UT failure. Diffet trunk vs. branch-3.2 diff and checked it thoroughly, LGTM. Committed your patch to branch-3.2. Resolving this jira. > Support listing of aggregated logs for containers belonging to an application > attempt > - > > Key: YARN-10101 > URL: https://issues.apache.org/jira/browse/YARN-10101 > Project: Hadoop YARN > Issue Type: Sub-task > Components: log-aggregation, yarn >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-10101.001.patch, YARN-10101.002.patch, > YARN-10101.003.patch, YARN-10101.004.patch, YARN-10101.005.patch, > YARN-10101.006.patch, YARN-10101.007.patch, YARN-10101.008.patch, > YARN-10101.009.patch, YARN-10101.branch-3.2.001.patch, > YARN-10101.branch-3.2.001.patch, YARN-10101.branch-3.2.002.patch > > > To display logs without access to the timeline server, we need an interface > where we can query the list of containers with aggregated logs belonging to > an application attempt. > We should add support for this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
