[
https://issues.apache.org/jira/browse/YARN-10082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014449#comment-17014449
]
Peter Bacsko commented on YARN-10082:
-------------------------------------
Attached patch v1 without tests.
> FS-CS converter: disable terminal placement rule checking
> ---------------------------------------------------------
>
> Key: YARN-10082
> URL: https://issues.apache.org/jira/browse/YARN-10082
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Peter Bacsko
> Assignee: Peter Bacsko
> Priority: Critical
> Attachments: YARN-10082-001.patch
>
>
> Before YARN-8967, {{QueuePlacementRule}} class had a method called
> {{isTerminal()}}. However, sometimes this method was hard-coded to return
> false, accepting such configurations as:
> {noformat}
> <queuePlacementPolicy>
> <rule name="specified" create="true"/>
> <rule name="nestedUserQueue" create="true">
> <rule name="default" create="true" queue="users"/>
> </rule>
> <rule name="default"/>
> </queuePlacementPolicy>
> {noformat}
> It's because {{NestedUserQueue.isTerminal()}} always returns {{false}}.
> This changed after YARN-8967. Now, this configuration is not accepted because
> {{QueuePlacementPolicy.fromXml()}} calculates a list of terminal rules
> differently:
> https://github.com/apache/hadoop/blob/5257f50abb71905ef3068fd45541d00ce9e8f355/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementPolicy.java#L176-L183
> In order to migrate existing configuration that were created before
> YARN-8967, we need a new switch (at least in migration mode) in FS to turn
> off this validation, otherwise the tool will not be able to migrate these
> configs and the following exception will be thrown:
> {noformat}
> ~$ ./yarn fs2cs -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp
> WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
> 20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Output
> directory for yarn-site.xml and capacity-scheduler.xml is: /tmp
> 20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Conversion
> rules file is not defined, using default conversion config!
> 20/01/13 05:48:21 INFO converter.FSConfigToCSConfigConverter: Using
> explicitly defined fair-scheduler.xml
> WARNING: This feature is experimental and not intended for production use!
> 20/01/13 05:48:21 INFO conf.Configuration: resource-types.xml not found
> 20/01/13 05:48:21 INFO resource.ResourceUtils: Unable to find
> 'resource-types.xml'.
> 20/01/13 05:48:21 INFO security.YarnAuthorizationProvider:
> org.apache.hadoop.yarn.security.ConfiguredYarnAuthorizer is instantiated.
> 20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Minimum allocation =
> <memory:1024, vCores:1>
> 20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Maximum allocation =
> <memory:8192, vCores:2>
> 20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
> 20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
> 20/01/13 05:48:21 INFO fair.AllocationFileLoaderService: Loading allocation
> file file:/tmp/fair-scheduler.xml
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
> 20/01/13 05:48:22 INFO service.AbstractService: Service
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
> failed in state INITED
> java.io.IOException: Failed to initialize FairScheduler
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1438)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1479)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:206)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:116)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:44)
> Caused by:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException:
> Rules after rule 2 in queue placement policy can never be reached
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.updateRuleSet(QueuePlacementPolicy.java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.fromXml(QueuePlacementPolicy.java:187)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.getQueuePlacementPolicy(AllocationFileLoaderService.java:276)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:257)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1436)
> ... 6 more
> 20/01/13 05:48:22 ERROR converter.FSConfigToCSConfigConverterMain: Error
> while starting FS configuration conversion!
> org.apache.hadoop.service.ServiceStateException: java.io.IOException: Failed
> to initialize FairScheduler
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:173)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:206)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:116)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:44)
> Caused by: java.io.IOException: Failed to initialize FairScheduler
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1438)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1479)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> ... 4 more
> Caused by:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException:
> Rules after rule 2 in queue placement policy can never be reached
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.updateRuleSet(QueuePlacementPolicy.java:110)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.fromXml(QueuePlacementPolicy.java:187)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.getQueuePlacementPolicy(AllocationFileLoaderService.java:276)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:257)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1436)
> ... 6 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
