[
https://issues.apache.org/jira/browse/YARN-10099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17026666#comment-17026666
]
Hadoop QA commented on YARN-10099:
----------------------------------
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 89m
17s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}147m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | YARN-10099 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12992217/YARN-10099-010.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 5de5f61301b4 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5977360 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_232 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25474/testReport/ |
| Max. process+thread count | 836 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25474/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> FS-CS converter: handle allow-undeclared-pools and user-as-default-queue
> properly and fix misc issues
> -----------------------------------------------------------------------------------------------------
>
> Key: YARN-10099
> URL: https://issues.apache.org/jira/browse/YARN-10099
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Peter Bacsko
> Assignee: Peter Bacsko
> Priority: Major
> Labels: fs2cs
> Attachments: YARN-10099-001.patch, YARN-10099-002.patch,
> YARN-10099-003.patch, YARN-10099-004.patch, YARN-10099-005.patch,
> YARN-10099-006.patch, YARN-10099-007.patch, YARN-10099-008.patch,
> YARN-10099-010.patch
>
>
> This ticket is intended to fix two issues:
> 1. Based on the latest documentation, there are two important properties that
> are ignored if we have placement rules:
> ||Property||Explanation||
> |yarn.scheduler.fair.allow-undeclared-pools|If this is true, new queues can
> be created at application submission time, whether because they are specified
> as the application’s queue by the submitter or because they are placed there
> by the user-as-default-queue property. If this is false, any time an app
> would be placed in a queue that is not specified in the allocations file, it
> is placed in the “default” queue instead. Defaults to true. *If a queue
> placement policy is given in the allocations file, this property is ignored.*|
> |yarn.scheduler.fair.user-as-default-queue|Whether to use the username
> associated with the allocation as the default queue name, in the event that a
> queue name is not specified. If this is set to “false” or unset, all jobs
> have a shared default queue, named “default”. Defaults to true. *If a queue
> placement policy is given in the allocations file, this property is ignored.*|
> Right now these settings affects the conversion regardless of the placement
> rules.
> 2. A converted configuration throws this error:
> {noformat}
> 2020-01-27 03:35:35,007 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned
> to standby state
> 2020-01-27 03:35:35,008 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.IllegalArgumentException: Illegal queue mapping
> u:%user:%user;u:%user:root.users.%user;u:%user:root.default
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueMappings(CapacitySchedulerConfiguration.java:1113)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.initialize(UserGroupMappingPlacementRule.java:244)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getUserGroupMappingPlacementRule(CapacityScheduler.java:671)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updatePlacementRules(CapacityScheduler.java:712)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:753)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> {noformat}
> Mapping rules should be separated by a "," character, not by a semicolon.
> 3. When initializing FS for conversion, we add the current {{yarn-site.xml}}
> as resource. This is not necessary. This can cause problems like:
> {noformat}
> [...]
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-audit.xml): couldn't find resource file
> location
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-security.xml): couldn't find resource file
> location
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-policymgr-ssl.xml): couldn't find resource
> file location
> 20/01/29 02:45:38 ERROR conf.Configuration: error parsing conf
> file:/etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml
> java.io.FileNotFoundException:
> /etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml (No such file or
> directory)
> at java.base/java.io.FileInputStream.open0(Native Method)
> at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
> at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
> at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112)
> at
> java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86)
> at
> java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:184)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2966)
> at
> org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3057)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3018)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2996)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2871)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1223)
> at
> org.apache.ranger.authorization.hadoop.config.RangerPluginConfig.addSslConfigResource(RangerPluginConfig.java:234)
> at
> org.apache.ranger.authorization.hadoop.config.RangerPluginConfig.addResourcesForServiceType(RangerPluginConfig.java:161)
> at
> org.apache.ranger.authorization.hadoop.config.RangerPluginConfig.<init>(RangerPluginConfig.java:51)
> at
> org.apache.ranger.plugin.service.RangerBasePlugin.<init>(RangerBasePlugin.java:75)
> at
> org.apache.ranger.authorization.yarn.authorizer.RangerYarnPlugin.<init>(RangerYarnAuthorizer.java:246)
> at
> org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:81)
> at
> org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:94)
> at
> org.apache.hadoop.yarn.security.YarnAuthorizationProvider.getInstance(YarnAuthorizationProvider.java:57)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1375)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1484)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:209)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:102)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:137)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:40)
> 20/01/29 02:45:38 WARN config.RangerPluginConfig: Unable to find SSL Configs
> 20/01/29 02:45:38 INFO config.RangerPluginConfig: PolicyEngineOptions: {
> evaluatorType: auto, evaluateDelegateAdminOnly: false,
> disableContextEnrichers: false, disableCustomConditions: false,
> disableTagPolicyEvaluation: false, enableTagEnricherWithLocalRefresher:
> false, disableTrieLookupPrefilter: false, optimizeTrieForRetrieval: false,
> cacheAuditResult: false }
> 20/01/29 02:45:38 INFO provider.AuditProviderFactory: AuditProviderFactory:
> creating..
> 20/01/29 02:45:38 INFO provider.AuditProviderFactory: AuditProviderFactory:
> initializing..
> 20/01/29 02:45:38 INFO provider.AuditProviderFactory: AuditProviderFactory:
> Audit not enabled..
> 20/01/29 02:45:38 INFO service.AbstractService: Service
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
> failed in state INITED
> java.lang.IllegalArgumentException: bound must be positive
> at java.base/java.util.Random.nextInt(Random.java:388)
> at
> org.apache.ranger.plugin.util.RangerRESTClient.<init>(RangerRESTClient.java:124)
> at
> org.apache.ranger.admin.client.RangerAdminRESTClient.init(RangerAdminRESTClient.java:765)
> at
> org.apache.ranger.admin.client.RangerAdminRESTClient.init(RangerAdminRESTClient.java:110)
> at
> org.apache.ranger.plugin.service.RangerBasePlugin.createAdminClient(RangerBasePlugin.java:584)
> at
> org.apache.ranger.plugin.util.PolicyRefresher.<init>(PolicyRefresher.java:95)
> at
> org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:176)
> at
> org.apache.ranger.authorization.yarn.authorizer.RangerYarnPlugin.init(RangerYarnAuthorizer.java:251)
> at
> org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:82)
> at
> org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:94)
> at
> org.apache.hadoop.yarn.security.YarnAuthorizationProvider.getInstance(YarnAuthorizationProvider.java:57)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1375)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1484)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:209)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:102)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:137)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:40)
> [...]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
