[jira] [Updated] (YARN-10099) FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues
[
https://issues.apache.org/jira/browse/YARN-10099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-10099:
--
Fix Version/s: 3.3.0
> FS-CS converter: handle allow-undeclared-pools and user-as-default-queue
> properly and fix misc issues
> -
>
> Key: YARN-10099
> URL: https://issues.apache.org/jira/browse/YARN-10099
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Fix For: 3.3.0
>
> Attachments: YARN-10099-001.patch, YARN-10099-002.patch,
> YARN-10099-003.patch, YARN-10099-004.patch, YARN-10099-005.patch,
> YARN-10099-006.patch, YARN-10099-007.patch, YARN-10099-008.patch,
> YARN-10099-010.patch
>
>
> This ticket is intended to fix two issues:
> 1. Based on the latest documentation, there are two important properties that
> are ignored if we have placement rules:
> ||Property||Explanation||
> |yarn.scheduler.fair.allow-undeclared-pools|If this is true, new queues can
> be created at application submission time, whether because they are specified
> as the application’s queue by the submitter or because they are placed there
> by the user-as-default-queue property. If this is false, any time an app
> would be placed in a queue that is not specified in the allocations file, it
> is placed in the “default” queue instead. Defaults to true. *If a queue
> placement policy is given in the allocations file, this property is ignored.*|
> |yarn.scheduler.fair.user-as-default-queue|Whether to use the username
> associated with the allocation as the default queue name, in the event that a
> queue name is not specified. If this is set to “false” or unset, all jobs
> have a shared default queue, named “default”. Defaults to true. *If a queue
> placement policy is given in the allocations file, this property is ignored.*|
> Right now these settings affects the conversion regardless of the placement
> rules.
> 2. A converted configuration throws this error:
> {noformat}
> 2020-01-27 03:35:35,007 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned
> to standby state
> 2020-01-27 03:35:35,008 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.IllegalArgumentException: Illegal queue mapping
> u:%user:%user;u:%user:root.users.%user;u:%user:root.default
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueMappings(CapacitySchedulerConfiguration.java:1113)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.initialize(UserGroupMappingPlacementRule.java:244)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getUserGroupMappingPlacementRule(CapacityScheduler.java:671)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updatePlacementRules(CapacityScheduler.java:712)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:753)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> {noformat}
> Mapping rules should be separated by a "," character, not by a semicolon.
> 3. When initializing FS for conversion, we add the current {{yarn-site.xml}}
> as resource. This is not necessary. This can cause problems like:
> {noformat}
> [...]
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-audit.xml): couldn't find resource file
> location
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-security.xml): couldn't find resource file
> location
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-policymgr-ssl.xml): couldn't find resource
> file location
> 20/01/29 02:45:38 ERROR conf.Configuration: error parsing conf
> file:/etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml
> java.io.FileNotFoundException:
> /etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml (No such file or
> directory)
> at java.base/java.io.FileInputStream.open0(Native Method)
> at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
>
[jira] [Updated] (YARN-10099) FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues
[
https://issues.apache.org/jira/browse/YARN-10099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10099:
Attachment: YARN-10099-010.patch
> FS-CS converter: handle allow-undeclared-pools and user-as-default-queue
> properly and fix misc issues
> -
>
> Key: YARN-10099
> URL: https://issues.apache.org/jira/browse/YARN-10099
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10099-001.patch, YARN-10099-002.patch,
> YARN-10099-003.patch, YARN-10099-004.patch, YARN-10099-005.patch,
> YARN-10099-006.patch, YARN-10099-007.patch, YARN-10099-008.patch,
> YARN-10099-010.patch
>
>
> This ticket is intended to fix two issues:
> 1. Based on the latest documentation, there are two important properties that
> are ignored if we have placement rules:
> ||Property||Explanation||
> |yarn.scheduler.fair.allow-undeclared-pools|If this is true, new queues can
> be created at application submission time, whether because they are specified
> as the application’s queue by the submitter or because they are placed there
> by the user-as-default-queue property. If this is false, any time an app
> would be placed in a queue that is not specified in the allocations file, it
> is placed in the “default” queue instead. Defaults to true. *If a queue
> placement policy is given in the allocations file, this property is ignored.*|
> |yarn.scheduler.fair.user-as-default-queue|Whether to use the username
> associated with the allocation as the default queue name, in the event that a
> queue name is not specified. If this is set to “false” or unset, all jobs
> have a shared default queue, named “default”. Defaults to true. *If a queue
> placement policy is given in the allocations file, this property is ignored.*|
> Right now these settings affects the conversion regardless of the placement
> rules.
> 2. A converted configuration throws this error:
> {noformat}
> 2020-01-27 03:35:35,007 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned
> to standby state
> 2020-01-27 03:35:35,008 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.IllegalArgumentException: Illegal queue mapping
> u:%user:%user;u:%user:root.users.%user;u:%user:root.default
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueMappings(CapacitySchedulerConfiguration.java:1113)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.initialize(UserGroupMappingPlacementRule.java:244)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getUserGroupMappingPlacementRule(CapacityScheduler.java:671)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updatePlacementRules(CapacityScheduler.java:712)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:753)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> {noformat}
> Mapping rules should be separated by a "," character, not by a semicolon.
> 3. When initializing FS for conversion, we add the current {{yarn-site.xml}}
> as resource. This is not necessary. This can cause problems like:
> {noformat}
> [...]
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-audit.xml): couldn't find resource file
> location
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-security.xml): couldn't find resource file
> location
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-policymgr-ssl.xml): couldn't find resource
> file location
> 20/01/29 02:45:38 ERROR conf.Configuration: error parsing conf
> file:/etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml
> java.io.FileNotFoundException:
> /etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml (No such file or
> directory)
> at java.base/java.io.FileInputStream.open0(Native Method)
> at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
> at
[jira] [Updated] (YARN-10099) FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues
[
https://issues.apache.org/jira/browse/YARN-10099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10099:
Attachment: YARN-10099-008.patch
> FS-CS converter: handle allow-undeclared-pools and user-as-default-queue
> properly and fix misc issues
> -
>
> Key: YARN-10099
> URL: https://issues.apache.org/jira/browse/YARN-10099
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10099-001.patch, YARN-10099-002.patch,
> YARN-10099-003.patch, YARN-10099-004.patch, YARN-10099-005.patch,
> YARN-10099-006.patch, YARN-10099-007.patch, YARN-10099-008.patch
>
>
> This ticket is intended to fix two issues:
> 1. Based on the latest documentation, there are two important properties that
> are ignored if we have placement rules:
> ||Property||Explanation||
> |yarn.scheduler.fair.allow-undeclared-pools|If this is true, new queues can
> be created at application submission time, whether because they are specified
> as the application’s queue by the submitter or because they are placed there
> by the user-as-default-queue property. If this is false, any time an app
> would be placed in a queue that is not specified in the allocations file, it
> is placed in the “default” queue instead. Defaults to true. *If a queue
> placement policy is given in the allocations file, this property is ignored.*|
> |yarn.scheduler.fair.user-as-default-queue|Whether to use the username
> associated with the allocation as the default queue name, in the event that a
> queue name is not specified. If this is set to “false” or unset, all jobs
> have a shared default queue, named “default”. Defaults to true. *If a queue
> placement policy is given in the allocations file, this property is ignored.*|
> Right now these settings affects the conversion regardless of the placement
> rules.
> 2. A converted configuration throws this error:
> {noformat}
> 2020-01-27 03:35:35,007 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned
> to standby state
> 2020-01-27 03:35:35,008 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.IllegalArgumentException: Illegal queue mapping
> u:%user:%user;u:%user:root.users.%user;u:%user:root.default
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueMappings(CapacitySchedulerConfiguration.java:1113)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.initialize(UserGroupMappingPlacementRule.java:244)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getUserGroupMappingPlacementRule(CapacityScheduler.java:671)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updatePlacementRules(CapacityScheduler.java:712)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:753)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> {noformat}
> Mapping rules should be separated by a "," character, not by a semicolon.
> 3. When initializing FS for conversion, we add the current {{yarn-site.xml}}
> as resource. This is not necessary. This can cause problems like:
> {noformat}
> [...]
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-audit.xml): couldn't find resource file
> location
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-security.xml): couldn't find resource file
> location
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-policymgr-ssl.xml): couldn't find resource
> file location
> 20/01/29 02:45:38 ERROR conf.Configuration: error parsing conf
> file:/etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml
> java.io.FileNotFoundException:
> /etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml (No such file or
> directory)
> at java.base/java.io.FileInputStream.open0(Native Method)
> at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
> at
[jira] [Updated] (YARN-10099) FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues
[
https://issues.apache.org/jira/browse/YARN-10099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10099:
Attachment: YARN-10099-007.patch
> FS-CS converter: handle allow-undeclared-pools and user-as-default-queue
> properly and fix misc issues
> -
>
> Key: YARN-10099
> URL: https://issues.apache.org/jira/browse/YARN-10099
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10099-001.patch, YARN-10099-002.patch,
> YARN-10099-003.patch, YARN-10099-004.patch, YARN-10099-005.patch,
> YARN-10099-006.patch, YARN-10099-007.patch
>
>
> This ticket is intended to fix two issues:
> 1. Based on the latest documentation, there are two important properties that
> are ignored if we have placement rules:
> ||Property||Explanation||
> |yarn.scheduler.fair.allow-undeclared-pools|If this is true, new queues can
> be created at application submission time, whether because they are specified
> as the application’s queue by the submitter or because they are placed there
> by the user-as-default-queue property. If this is false, any time an app
> would be placed in a queue that is not specified in the allocations file, it
> is placed in the “default” queue instead. Defaults to true. *If a queue
> placement policy is given in the allocations file, this property is ignored.*|
> |yarn.scheduler.fair.user-as-default-queue|Whether to use the username
> associated with the allocation as the default queue name, in the event that a
> queue name is not specified. If this is set to “false” or unset, all jobs
> have a shared default queue, named “default”. Defaults to true. *If a queue
> placement policy is given in the allocations file, this property is ignored.*|
> Right now these settings affects the conversion regardless of the placement
> rules.
> 2. A converted configuration throws this error:
> {noformat}
> 2020-01-27 03:35:35,007 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned
> to standby state
> 2020-01-27 03:35:35,008 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.IllegalArgumentException: Illegal queue mapping
> u:%user:%user;u:%user:root.users.%user;u:%user:root.default
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueMappings(CapacitySchedulerConfiguration.java:1113)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.initialize(UserGroupMappingPlacementRule.java:244)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getUserGroupMappingPlacementRule(CapacityScheduler.java:671)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updatePlacementRules(CapacityScheduler.java:712)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:753)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> {noformat}
> Mapping rules should be separated by a "," character, not by a semicolon.
> 3. When initializing FS for conversion, we add the current {{yarn-site.xml}}
> as resource. This is not necessary. This can cause problems like:
> {noformat}
> [...]
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-audit.xml): couldn't find resource file
> location
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-security.xml): couldn't find resource file
> location
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-policymgr-ssl.xml): couldn't find resource
> file location
> 20/01/29 02:45:38 ERROR conf.Configuration: error parsing conf
> file:/etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml
> java.io.FileNotFoundException:
> /etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml (No such file or
> directory)
> at java.base/java.io.FileInputStream.open0(Native Method)
> at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
> at java.base/java.io.FileInputStream.(FileInputStream.java:157)
>
[jira] [Updated] (YARN-10099) FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues
[
https://issues.apache.org/jira/browse/YARN-10099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10099:
Attachment: YARN-10099-006.patch
> FS-CS converter: handle allow-undeclared-pools and user-as-default-queue
> properly and fix misc issues
> -
>
> Key: YARN-10099
> URL: https://issues.apache.org/jira/browse/YARN-10099
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10099-001.patch, YARN-10099-002.patch,
> YARN-10099-003.patch, YARN-10099-004.patch, YARN-10099-005.patch,
> YARN-10099-006.patch
>
>
> This ticket is intended to fix two issues:
> 1. Based on the latest documentation, there are two important properties that
> are ignored if we have placement rules:
> ||Property||Explanation||
> |yarn.scheduler.fair.allow-undeclared-pools|If this is true, new queues can
> be created at application submission time, whether because they are specified
> as the application’s queue by the submitter or because they are placed there
> by the user-as-default-queue property. If this is false, any time an app
> would be placed in a queue that is not specified in the allocations file, it
> is placed in the “default” queue instead. Defaults to true. *If a queue
> placement policy is given in the allocations file, this property is ignored.*|
> |yarn.scheduler.fair.user-as-default-queue|Whether to use the username
> associated with the allocation as the default queue name, in the event that a
> queue name is not specified. If this is set to “false” or unset, all jobs
> have a shared default queue, named “default”. Defaults to true. *If a queue
> placement policy is given in the allocations file, this property is ignored.*|
> Right now these settings affects the conversion regardless of the placement
> rules.
> 2. A converted configuration throws this error:
> {noformat}
> 2020-01-27 03:35:35,007 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned
> to standby state
> 2020-01-27 03:35:35,008 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.IllegalArgumentException: Illegal queue mapping
> u:%user:%user;u:%user:root.users.%user;u:%user:root.default
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueMappings(CapacitySchedulerConfiguration.java:1113)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.initialize(UserGroupMappingPlacementRule.java:244)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getUserGroupMappingPlacementRule(CapacityScheduler.java:671)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updatePlacementRules(CapacityScheduler.java:712)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:753)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> {noformat}
> Mapping rules should be separated by a "," character, not by a semicolon.
> 3. When initializing FS for conversion, we add the current {{yarn-site.xml}}
> as resource. This is not necessary. This can cause problems like:
> {noformat}
> [...]
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-audit.xml): couldn't find resource file
> location
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-security.xml): couldn't find resource file
> location
> 20/01/29 02:45:38 ERROR config.RangerConfiguration:
> addResourceIfReadable(ranger-yarn-policymgr-ssl.xml): couldn't find resource
> file location
> 20/01/29 02:45:38 ERROR conf.Configuration: error parsing conf
> file:/etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml
> java.io.FileNotFoundException:
> /etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml (No such file or
> directory)
> at java.base/java.io.FileInputStream.open0(Native Method)
> at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
> at java.base/java.io.FileInputStream.(FileInputStream.java:157)
> at
[jira] [Updated] (YARN-10099) FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues
[
https://issues.apache.org/jira/browse/YARN-10099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10099:
Description:
This ticket is intended to fix two issues:
1. Based on the latest documentation, there are two important properties that
are ignored if we have placement rules:
||Property||Explanation||
|yarn.scheduler.fair.allow-undeclared-pools|If this is true, new queues can be
created at application submission time, whether because they are specified as
the application’s queue by the submitter or because they are placed there by
the user-as-default-queue property. If this is false, any time an app would be
placed in a queue that is not specified in the allocations file, it is placed
in the “default” queue instead. Defaults to true. *If a queue placement policy
is given in the allocations file, this property is ignored.*|
|yarn.scheduler.fair.user-as-default-queue|Whether to use the username
associated with the allocation as the default queue name, in the event that a
queue name is not specified. If this is set to “false” or unset, all jobs have
a shared default queue, named “default”. Defaults to true. *If a queue
placement policy is given in the allocations file, this property is ignored.*|
Right now these settings affects the conversion regardless of the placement
rules.
2. A converted configuration throws this error:
{noformat}
2020-01-27 03:35:35,007 INFO
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to
standby state
2020-01-27 03:35:35,008 FATAL
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
ResourceManager
java.lang.IllegalArgumentException: Illegal queue mapping
u:%user:%user;u:%user:root.users.%user;u:%user:root.default
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueMappings(CapacitySchedulerConfiguration.java:1113)
at
org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.initialize(UserGroupMappingPlacementRule.java:244)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getUserGroupMappingPlacementRule(CapacityScheduler.java:671)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updatePlacementRules(CapacityScheduler.java:712)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:753)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
{noformat}
Mapping rules should be separated by a "," character, not by a semicolon.
3. When initializing FS for conversion, we add the current {{yarn-site.xml}} as
resource. This is not necessary. This can cause problems like:
{noformat}
[...]
20/01/29 02:45:38 ERROR config.RangerConfiguration:
addResourceIfReadable(ranger-yarn-audit.xml): couldn't find resource file
location
20/01/29 02:45:38 ERROR config.RangerConfiguration:
addResourceIfReadable(ranger-yarn-security.xml): couldn't find resource file
location
20/01/29 02:45:38 ERROR config.RangerConfiguration:
addResourceIfReadable(ranger-yarn-policymgr-ssl.xml): couldn't find resource
file location
20/01/29 02:45:38 ERROR conf.Configuration: error parsing conf
file:/etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml
java.io.FileNotFoundException:
/etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.(FileInputStream.java:157)
at java.base/java.io.FileInputStream.(FileInputStream.java:112)
at
java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86)
at
java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:184)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2966)
at
org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3057)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3018)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2996)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2871)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1223)
at
[jira] [Updated] (YARN-10099) FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues
[
https://issues.apache.org/jira/browse/YARN-10099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10099:
Description:
This ticket is intended to fix two issues:
1. Based on the latest documentation, there are two important properties that
are ignored if we have placement rules:
||Property||Explanation||
|yarn.scheduler.fair.allow-undeclared-pools|If this is true, new queues can be
created at application submission time, whether because they are specified as
the application’s queue by the submitter or because they are placed there by
the user-as-default-queue property. If this is false, any time an app would be
placed in a queue that is not specified in the allocations file, it is placed
in the “default” queue instead. Defaults to true. *If a queue placement policy
is given in the allocations file, this property is ignored.*|
|yarn.scheduler.fair.user-as-default-queue|Whether to use the username
associated with the allocation as the default queue name, in the event that a
queue name is not specified. If this is set to “false” or unset, all jobs have
a shared default queue, named “default”. Defaults to true. *If a queue
placement policy is given in the allocations file, this property is ignored.*|
Right now these settings affects the conversion regardless of the placement
rules.
2. A converted configuration throws this error:
{noformat}
2020-01-27 03:35:35,007 INFO
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to
standby state
2020-01-27 03:35:35,008 FATAL
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
ResourceManager
java.lang.IllegalArgumentException: Illegal queue mapping
u:%user:%user;u:%user:root.users.%user;u:%user:root.default
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueMappings(CapacitySchedulerConfiguration.java:1113)
at
org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.initialize(UserGroupMappingPlacementRule.java:244)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getUserGroupMappingPlacementRule(CapacityScheduler.java:671)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updatePlacementRules(CapacityScheduler.java:712)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:753)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
{noformat}
Mapping rules should be separated by a "," character, not by a semicolon.
3. When initializing FS for conversion, we add the current {{yarn-site.xml}} as
resource. This is not necessary. This can cause problems like:
{noformat}
[...]
1.cdh7.1.1.p0.1825944/lib/hadoop/lib/ranger-yarn-plugin-impl/gethostname4j-0.0.2.jar
20/01/29 02:45:38 ERROR config.RangerConfiguration:
addResourceIfReadable(ranger-yarn-audit.xml): couldn't find resource file
location
20/01/29 02:45:38 ERROR config.RangerConfiguration:
addResourceIfReadable(ranger-yarn-security.xml): couldn't find resource file
location
20/01/29 02:45:38 ERROR config.RangerConfiguration:
addResourceIfReadable(ranger-yarn-policymgr-ssl.xml): couldn't find resource
file location
20/01/29 02:45:38 ERROR conf.Configuration: error parsing conf
file:/etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml
java.io.FileNotFoundException:
/etc/hadoop/conf.cloudera.YARN-1/xasecure-audit.xml (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.(FileInputStream.java:157)
at java.base/java.io.FileInputStream.(FileInputStream.java:112)
at
java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86)
at
java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:184)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2966)
at
org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3057)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3018)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2996)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2871)
at
[jira] [Updated] (YARN-10099) FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues
[
https://issues.apache.org/jira/browse/YARN-10099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10099:
Summary: FS-CS converter: handle allow-undeclared-pools and
user-as-default-queue properly and fix misc issues (was: FS-CS converter:
handle allow-undeclared-pools and user-as-default-queue properly and fix
mapping rule separator)
> FS-CS converter: handle allow-undeclared-pools and user-as-default-queue
> properly and fix misc issues
> -
>
> Key: YARN-10099
> URL: https://issues.apache.org/jira/browse/YARN-10099
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10099-001.patch, YARN-10099-002.patch,
> YARN-10099-003.patch, YARN-10099-004.patch, YARN-10099-005.patch
>
>
> This ticket is intended to fix two issues:
> 1. Based on the latest documentation, there are two important properties that
> are ignored if we have placement rules:
> ||Property||Explanation||
> |yarn.scheduler.fair.allow-undeclared-pools|If this is true, new queues can
> be created at application submission time, whether because they are specified
> as the application’s queue by the submitter or because they are placed there
> by the user-as-default-queue property. If this is false, any time an app
> would be placed in a queue that is not specified in the allocations file, it
> is placed in the “default” queue instead. Defaults to true. *If a queue
> placement policy is given in the allocations file, this property is ignored.*|
> |yarn.scheduler.fair.user-as-default-queue|Whether to use the username
> associated with the allocation as the default queue name, in the event that a
> queue name is not specified. If this is set to “false” or unset, all jobs
> have a shared default queue, named “default”. Defaults to true. *If a queue
> placement policy is given in the allocations file, this property is ignored.*|
> Right now these settings affects the conversion regardless of the placement
> rules.
> 2. A converted configuration throws this error:
> {noformat}
> 2020-01-27 03:35:35,007 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned
> to standby state
> 2020-01-27 03:35:35,008 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.IllegalArgumentException: Illegal queue mapping
> u:%user:%user;u:%user:root.users.%user;u:%user:root.default
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueMappings(CapacitySchedulerConfiguration.java:1113)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.initialize(UserGroupMappingPlacementRule.java:244)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getUserGroupMappingPlacementRule(CapacityScheduler.java:671)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updatePlacementRules(CapacityScheduler.java:712)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:753)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> {noformat}
> Mapping rules should be separated by a "," character, not by a semicolon.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
