[ https://issues.apache.org/jira/browse/YARN-10099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17026649#comment-17026649 ]
Hadoop QA commented on YARN-10099: ---------------------------------- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 1s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 58s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 20s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 84m 4s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}135m 40s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 | | JIRA Issue | YARN-10099 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12992215/YARN-10099-008.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 7ce0838e0466 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 5977360 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_232 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/25473/testReport/ | | Max. process+thread count | 887 (vs. ulimit of 5500) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/25473/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > FS-CS converter: handle allow-undeclared-pools and user-as-default-queue > properly and fix misc issues > ----------------------------------------------------------------------------------------------------- > > Key: YARN-10099 > URL: https://issues.apache.org/jira/browse/YARN-10099 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Peter Bacsko > Assignee: Peter Bacsko > Priority: Major > Labels: fs2cs > Attachments: YARN-10099-001.patch, YARN-10099-002.patch, > YARN-10099-003.patch, YARN-10099-004.patch, YARN-10099-005.patch, > YARN-10099-006.patch, YARN-10099-007.patch, YARN-10099-008.patch, > YARN-10099-010.patch > > > This ticket is intended to fix two issues: > 1. Based on the latest documentation, there are two important properties that > are ignored if we have placement rules: > ||Property||Explanation|| > |yarn.scheduler.fair.allow-undeclared-pools|If this is true, new queues can > be created at application submission time, whether because they are specified > as the application’s queue by the submitter or because they are placed there > by the user-as-default-queue property. If this is false, any time an app > would be placed in a queue that is not specified in the allocations file, it > is placed in the “default” queue instead. Defaults to true. *If a queue > placement policy is given in the allocations file, this property is ignored.*| > |yarn.scheduler.fair.user-as-default-queue|Whether to use the username > associated with the allocation as the default queue name, in the event that a > queue name is not specified. If this is set to “false” or unset, all jobs > have a shared default queue, named “default”. Defaults to true. *If a queue > placement policy is given in the allocations file, this property is ignored.*| > Right now these settings affects the conversion regardless of the placement > rules. > 2. A converted configuration throws this error: > {noformat} > 2020-01-27 03:35:35,007 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned > to standby state > 2020-01-27 03:35:35,008 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting > ResourceManager > java.lang.IllegalArgumentException: Illegal queue mapping > u:%user:%user;u:%user:root.users.%user;u:%user:root.default > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueMappings(CapacitySchedulerConfiguration.java:1113) > at > org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.initialize(UserGroupMappingPlacementRule.java:244) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getUserGroupMappingPlacementRule(CapacityScheduler.java:671) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updatePlacementRules(CapacityScheduler.java:712) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:753) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) > {noformat} > Mapping rules should be separated by a "," character, not by a semicolon. > 3. When initializing FS for conversion, we add the current {{yarn-site.xml}} > as resource. This is not necessary. This can cause problems like: > {noformat} > [...] > 20/01/29 02:45:38 ERROR config.RangerConfiguration: > addResourceIfReadable(ranger-yarn-audit.xml): couldn't find resource file > location > 20/01/29 02:45:38 ERROR config.RangerConfiguration: > addResourceIfReadable(ranger-yarn-security.xml): couldn't find resource file > location > 20/01/29 02:45:38 ERROR config.RangerConfiguration: > addResourceIfReadable(ranger-yarn-policymgr-ssl.xml): couldn't find resource > file location > 20/01/29 02:45:38 ERROR conf.Configuration: error parsing conf > file:/etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml > java.io.FileNotFoundException: > /etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml (No such file or > directory) > at java.base/java.io.FileInputStream.open0(Native Method) > at java.base/java.io.FileInputStream.open(FileInputStream.java:219) > at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157) > at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112) > at > java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86) > at > java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:184) > at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2966) > at > org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3057) > at > org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3018) > at > org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2996) > at > org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2871) > at org.apache.hadoop.conf.Configuration.get(Configuration.java:1223) > at > org.apache.ranger.authorization.hadoop.config.RangerPluginConfig.addSslConfigResource(RangerPluginConfig.java:234) > at > org.apache.ranger.authorization.hadoop.config.RangerPluginConfig.addResourcesForServiceType(RangerPluginConfig.java:161) > at > org.apache.ranger.authorization.hadoop.config.RangerPluginConfig.<init>(RangerPluginConfig.java:51) > at > org.apache.ranger.plugin.service.RangerBasePlugin.<init>(RangerBasePlugin.java:75) > at > org.apache.ranger.authorization.yarn.authorizer.RangerYarnPlugin.<init>(RangerYarnAuthorizer.java:246) > at > org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:81) > at > org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:94) > at > org.apache.hadoop.yarn.security.YarnAuthorizationProvider.getInstance(YarnAuthorizationProvider.java:57) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1484) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:209) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:102) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:137) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:40) > 20/01/29 02:45:38 WARN config.RangerPluginConfig: Unable to find SSL Configs > 20/01/29 02:45:38 INFO config.RangerPluginConfig: PolicyEngineOptions: { > evaluatorType: auto, evaluateDelegateAdminOnly: false, > disableContextEnrichers: false, disableCustomConditions: false, > disableTagPolicyEvaluation: false, enableTagEnricherWithLocalRefresher: > false, disableTrieLookupPrefilter: false, optimizeTrieForRetrieval: false, > cacheAuditResult: false } > 20/01/29 02:45:38 INFO provider.AuditProviderFactory: AuditProviderFactory: > creating.. > 20/01/29 02:45:38 INFO provider.AuditProviderFactory: AuditProviderFactory: > initializing.. > 20/01/29 02:45:38 INFO provider.AuditProviderFactory: AuditProviderFactory: > Audit not enabled.. > 20/01/29 02:45:38 INFO service.AbstractService: Service > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler > failed in state INITED > java.lang.IllegalArgumentException: bound must be positive > at java.base/java.util.Random.nextInt(Random.java:388) > at > org.apache.ranger.plugin.util.RangerRESTClient.<init>(RangerRESTClient.java:124) > at > org.apache.ranger.admin.client.RangerAdminRESTClient.init(RangerAdminRESTClient.java:765) > at > org.apache.ranger.admin.client.RangerAdminRESTClient.init(RangerAdminRESTClient.java:110) > at > org.apache.ranger.plugin.service.RangerBasePlugin.createAdminClient(RangerBasePlugin.java:584) > at > org.apache.ranger.plugin.util.PolicyRefresher.<init>(PolicyRefresher.java:95) > at > org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:176) > at > org.apache.ranger.authorization.yarn.authorizer.RangerYarnPlugin.init(RangerYarnAuthorizer.java:251) > at > org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:82) > at > org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:94) > at > org.apache.hadoop.yarn.security.YarnAuthorizationProvider.getInstance(YarnAuthorizationProvider.java:57) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1375) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1484) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:209) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:102) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:137) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:40) > [...] > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org