[jira] [Updated] (YARN-10458) Hive On Tez queries fails upon submission to dynamically created pools
[
https://issues.apache.org/jira/browse/YARN-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10458:
Fix Version/s: 3.3.1
> Hive On Tez queries fails upon submission to dynamically created pools
> --
>
> Key: YARN-10458
> URL: https://issues.apache.org/jira/browse/YARN-10458
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Anand Srinivasan
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.4.0, 3.3.1
>
> Attachments: YARN-10458-001.patch, YARN-10458-002.patch,
> YARN-10458-003.patch, YARN-10458-004.patch, YARN-10458-branch-3.3.001.patch
>
>
> While using Dynamic Auto-Creation and Management of Leaf Queues, we could see
> that the queue creation fails because ACL submit application check couldn't
> succeed.
> We tried setting acl_submit_applications to '*' for managed parent queues.
> For static queues, this worked but failed for dynamic queues. Also tried
> setting the below property but it didn't help either.
> yarn.scheduler.capacity.root.parent-queue-name.leaf-queue-template.acl_submit_applications=*.
> RM error log shows the following :
> 2020-09-18 01:08:40,579 INFO
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule:
> Application application_1600399068816_0460 user user1 mapping [default] to
> [queue1] override false
> 2020-09-18 01:08:40,579 WARN
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: User 'user1' from
> application tag does not have access to queue 'user1'. The placement is done
> for user 'hive'
>
> Checking the code, scheduler#checkAccess() bails out even before checking the
> ACL permissions for that particular queue because the CSQueue is null.
> {code:java}
> public boolean checkAccess(UserGroupInformation callerUGI,
> QueueACL acl, String queueName) {
> CSQueue queue = getQueue(queueName);
> if (queue == null) {
> if (LOG.isDebugEnabled())
> { LOG.debug("ACL not found for queue access-type " + acl + " for queue " +
> queueName); }
> return false;*<-- the method returns false here.*
> }
> return queue.hasAccess(acl, callerUGI);
> }
> {code}
> As this is an auto created queue, CSQueue may be null in this case. May be
> scheduler#checkAccess() should have a logic to differentiate when CSQueue is
> null and if queue mapping is involved and if so, check if the parent queue
> exists and is a managed parent and if so, check if the parent queue has valid
> ACL's instead of returning false ?
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10458) Hive On Tez queries fails upon submission to dynamically created pools
[
https://issues.apache.org/jira/browse/YARN-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10458:
Attachment: YARN-10458-branch-3.3.001.patch
> Hive On Tez queries fails upon submission to dynamically created pools
> --
>
> Key: YARN-10458
> URL: https://issues.apache.org/jira/browse/YARN-10458
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Anand Srinivasan
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10458-001.patch, YARN-10458-002.patch,
> YARN-10458-003.patch, YARN-10458-004.patch, YARN-10458-branch-3.3.001.patch
>
>
> While using Dynamic Auto-Creation and Management of Leaf Queues, we could see
> that the queue creation fails because ACL submit application check couldn't
> succeed.
> We tried setting acl_submit_applications to '*' for managed parent queues.
> For static queues, this worked but failed for dynamic queues. Also tried
> setting the below property but it didn't help either.
> yarn.scheduler.capacity.root.parent-queue-name.leaf-queue-template.acl_submit_applications=*.
> RM error log shows the following :
> 2020-09-18 01:08:40,579 INFO
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule:
> Application application_1600399068816_0460 user user1 mapping [default] to
> [queue1] override false
> 2020-09-18 01:08:40,579 WARN
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: User 'user1' from
> application tag does not have access to queue 'user1'. The placement is done
> for user 'hive'
>
> Checking the code, scheduler#checkAccess() bails out even before checking the
> ACL permissions for that particular queue because the CSQueue is null.
> {code:java}
> public boolean checkAccess(UserGroupInformation callerUGI,
> QueueACL acl, String queueName) {
> CSQueue queue = getQueue(queueName);
> if (queue == null) {
> if (LOG.isDebugEnabled())
> { LOG.debug("ACL not found for queue access-type " + acl + " for queue " +
> queueName); }
> return false;*<-- the method returns false here.*
> }
> return queue.hasAccess(acl, callerUGI);
> }
> {code}
> As this is an auto created queue, CSQueue may be null in this case. May be
> scheduler#checkAccess() should have a logic to differentiate when CSQueue is
> null and if queue mapping is involved and if so, check if the parent queue
> exists and is a managed parent and if so, check if the parent queue has valid
> ACL's instead of returning false ?
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10458) Hive On Tez queries fails upon submission to dynamically created pools
[
https://issues.apache.org/jira/browse/YARN-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-10458:
--
Fix Version/s: 3.4.0
> Hive On Tez queries fails upon submission to dynamically created pools
> --
>
> Key: YARN-10458
> URL: https://issues.apache.org/jira/browse/YARN-10458
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Anand Srinivasan
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10458-001.patch, YARN-10458-002.patch,
> YARN-10458-003.patch, YARN-10458-004.patch
>
>
> While using Dynamic Auto-Creation and Management of Leaf Queues, we could see
> that the queue creation fails because ACL submit application check couldn't
> succeed.
> We tried setting acl_submit_applications to '*' for managed parent queues.
> For static queues, this worked but failed for dynamic queues. Also tried
> setting the below property but it didn't help either.
> yarn.scheduler.capacity.root.parent-queue-name.leaf-queue-template.acl_submit_applications=*.
> RM error log shows the following :
> 2020-09-18 01:08:40,579 INFO
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule:
> Application application_1600399068816_0460 user user1 mapping [default] to
> [queue1] override false
> 2020-09-18 01:08:40,579 WARN
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: User 'user1' from
> application tag does not have access to queue 'user1'. The placement is done
> for user 'hive'
>
> Checking the code, scheduler#checkAccess() bails out even before checking the
> ACL permissions for that particular queue because the CSQueue is null.
> {code:java}
> public boolean checkAccess(UserGroupInformation callerUGI,
> QueueACL acl, String queueName) {
> CSQueue queue = getQueue(queueName);
> if (queue == null) {
> if (LOG.isDebugEnabled())
> { LOG.debug("ACL not found for queue access-type " + acl + " for queue " +
> queueName); }
> return false;*<-- the method returns false here.*
> }
> return queue.hasAccess(acl, callerUGI);
> }
> {code}
> As this is an auto created queue, CSQueue may be null in this case. May be
> scheduler#checkAccess() should have a logic to differentiate when CSQueue is
> null and if queue mapping is involved and if so, check if the parent queue
> exists and is a managed parent and if so, check if the parent queue has valid
> ACL's instead of returning false ?
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10458) Hive On Tez queries fails upon submission to dynamically created pools
[
https://issues.apache.org/jira/browse/YARN-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10458:
Attachment: YARN-10458-004.patch
> Hive On Tez queries fails upon submission to dynamically created pools
> --
>
> Key: YARN-10458
> URL: https://issues.apache.org/jira/browse/YARN-10458
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Anand Srinivasan
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10458-001.patch, YARN-10458-002.patch,
> YARN-10458-003.patch, YARN-10458-004.patch
>
>
> While using Dynamic Auto-Creation and Management of Leaf Queues, we could see
> that the queue creation fails because ACL submit application check couldn't
> succeed.
> We tried setting acl_submit_applications to '*' for managed parent queues.
> For static queues, this worked but failed for dynamic queues. Also tried
> setting the below property but it didn't help either.
> yarn.scheduler.capacity.root.parent-queue-name.leaf-queue-template.acl_submit_applications=*.
> RM error log shows the following :
> 2020-09-18 01:08:40,579 INFO
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule:
> Application application_1600399068816_0460 user user1 mapping [default] to
> [queue1] override false
> 2020-09-18 01:08:40,579 WARN
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: User 'user1' from
> application tag does not have access to queue 'user1'. The placement is done
> for user 'hive'
>
> Checking the code, scheduler#checkAccess() bails out even before checking the
> ACL permissions for that particular queue because the CSQueue is null.
> {code:java}
> public boolean checkAccess(UserGroupInformation callerUGI,
> QueueACL acl, String queueName) {
> CSQueue queue = getQueue(queueName);
> if (queue == null) {
> if (LOG.isDebugEnabled())
> { LOG.debug("ACL not found for queue access-type " + acl + " for queue " +
> queueName); }
> return false;*<-- the method returns false here.*
> }
> return queue.hasAccess(acl, callerUGI);
> }
> {code}
> As this is an auto created queue, CSQueue may be null in this case. May be
> scheduler#checkAccess() should have a logic to differentiate when CSQueue is
> null and if queue mapping is involved and if so, check if the parent queue
> exists and is a managed parent and if so, check if the parent queue has valid
> ACL's instead of returning false ?
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10458) Hive On Tez queries fails upon submission to dynamically created pools
[
https://issues.apache.org/jira/browse/YARN-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10458:
Attachment: YARN-10458-003.patch
> Hive On Tez queries fails upon submission to dynamically created pools
> --
>
> Key: YARN-10458
> URL: https://issues.apache.org/jira/browse/YARN-10458
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Anand Srinivasan
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10458-001.patch, YARN-10458-002.patch,
> YARN-10458-003.patch
>
>
> While using Dynamic Auto-Creation and Management of Leaf Queues, we could see
> that the queue creation fails because ACL submit application check couldn't
> succeed.
> We tried setting acl_submit_applications to '*' for managed parent queues.
> For static queues, this worked but failed for dynamic queues. Also tried
> setting the below property but it didn't help either.
> yarn.scheduler.capacity.root.parent-queue-name.leaf-queue-template.acl_submit_applications=*.
> RM error log shows the following :
> 2020-09-18 01:08:40,579 INFO
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule:
> Application application_1600399068816_0460 user user1 mapping [default] to
> [queue1] override false
> 2020-09-18 01:08:40,579 WARN
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: User 'user1' from
> application tag does not have access to queue 'user1'. The placement is done
> for user 'hive'
>
> Checking the code, scheduler#checkAccess() bails out even before checking the
> ACL permissions for that particular queue because the CSQueue is null.
> {code:java}
> public boolean checkAccess(UserGroupInformation callerUGI,
> QueueACL acl, String queueName) {
> CSQueue queue = getQueue(queueName);
> if (queue == null) {
> if (LOG.isDebugEnabled())
> { LOG.debug("ACL not found for queue access-type " + acl + " for queue " +
> queueName); }
> return false;*<-- the method returns false here.*
> }
> return queue.hasAccess(acl, callerUGI);
> }
> {code}
> As this is an auto created queue, CSQueue may be null in this case. May be
> scheduler#checkAccess() should have a logic to differentiate when CSQueue is
> null and if queue mapping is involved and if so, check if the parent queue
> exists and is a managed parent and if so, check if the parent queue has valid
> ACL's instead of returning false ?
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10458) Hive On Tez queries fails upon submission to dynamically created pools
[
https://issues.apache.org/jira/browse/YARN-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10458:
Attachment: YARN-10458-002.patch
> Hive On Tez queries fails upon submission to dynamically created pools
> --
>
> Key: YARN-10458
> URL: https://issues.apache.org/jira/browse/YARN-10458
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Anand Srinivasan
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10458-001.patch, YARN-10458-002.patch
>
>
> While using Dynamic Auto-Creation and Management of Leaf Queues, we could see
> that the queue creation fails because ACL submit application check couldn't
> succeed.
> We tried setting acl_submit_applications to '*' for managed parent queues.
> For static queues, this worked but failed for dynamic queues. Also tried
> setting the below property but it didn't help either.
> yarn.scheduler.capacity.root.parent-queue-name.leaf-queue-template.acl_submit_applications=*.
> RM error log shows the following :
> 2020-09-18 01:08:40,579 INFO
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule:
> Application application_1600399068816_0460 user user1 mapping [default] to
> [queue1] override false
> 2020-09-18 01:08:40,579 WARN
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: User 'user1' from
> application tag does not have access to queue 'user1'. The placement is done
> for user 'hive'
>
> Checking the code, scheduler#checkAccess() bails out even before checking the
> ACL permissions for that particular queue because the CSQueue is null.
> {code:java}
> public boolean checkAccess(UserGroupInformation callerUGI,
> QueueACL acl, String queueName) {
> CSQueue queue = getQueue(queueName);
> if (queue == null) {
> if (LOG.isDebugEnabled())
> { LOG.debug("ACL not found for queue access-type " + acl + " for queue " +
> queueName); }
> return false;*<-- the method returns false here.*
> }
> return queue.hasAccess(acl, callerUGI);
> }
> {code}
> As this is an auto created queue, CSQueue may be null in this case. May be
> scheduler#checkAccess() should have a logic to differentiate when CSQueue is
> null and if queue mapping is involved and if so, check if the parent queue
> exists and is a managed parent and if so, check if the parent queue has valid
> ACL's instead of returning false ?
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10458) Hive On Tez queries fails upon submission to dynamically created pools
[
https://issues.apache.org/jira/browse/YARN-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10458:
Attachment: YARN-10458-001.patch
> Hive On Tez queries fails upon submission to dynamically created pools
> --
>
> Key: YARN-10458
> URL: https://issues.apache.org/jira/browse/YARN-10458
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Anand Srinivasan
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10458-001.patch
>
>
> While using Dynamic Auto-Creation and Management of Leaf Queues, we could see
> that the queue creation fails because ACL submit application check couldn't
> succeed.
> We tried setting acl_submit_applications to '*' for managed parent queues.
> For static queues, this worked but failed for dynamic queues. Also tried
> setting the below property but it didn't help either.
> yarn.scheduler.capacity.root.parent-queue-name.leaf-queue-template.acl_submit_applications=*.
> RM error log shows the following :
> 2020-09-18 01:08:40,579 INFO
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule:
> Application application_1600399068816_0460 user user1 mapping [default] to
> [queue1] override false
> 2020-09-18 01:08:40,579 WARN
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: User 'user1' from
> application tag does not have access to queue 'user1'. The placement is done
> for user 'hive'
>
> Checking the code, scheduler#checkAccess() bails out even before checking the
> ACL permissions for that particular queue because the CSQueue is null.
> {code:java}
> public boolean checkAccess(UserGroupInformation callerUGI,
> QueueACL acl, String queueName) {
> CSQueue queue = getQueue(queueName);
> if (queue == null) {
> if (LOG.isDebugEnabled())
> { LOG.debug("ACL not found for queue access-type " + acl + " for queue " +
> queueName); }
> return false;*<-- the method returns false here.*
> }
> return queue.hasAccess(acl, callerUGI);
> }
> {code}
> As this is an auto created queue, CSQueue may be null in this case. May be
> scheduler#checkAccess() should have a logic to differentiate when CSQueue is
> null and if queue mapping is involved and if so, check if the parent queue
> exists and is a managed parent and if so, check if the parent queue has valid
> ACL's instead of returning false ?
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10458) Hive On Tez queries fails upon submission to dynamically created pools
[
https://issues.apache.org/jira/browse/YARN-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tarun Parimi updated YARN-10458:
Description:
While using Dynamic Auto-Creation and Management of Leaf Queues, we could see
that the queue creation fails because ACL submit application check couldn't
succeed.
We tried setting acl_submit_applications to '*' for managed parent queues. For
static queues, this worked but failed for dynamic queues. Also tried setting
the below property but it didn't help either.
yarn.scheduler.capacity.root.parent-queue-name.leaf-queue-template.acl_submit_applications=*.
RM error log shows the following :
2020-09-18 01:08:40,579 INFO
org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule:
Application application_1600399068816_0460 user user1 mapping [default] to
[queue1] override false
2020-09-18 01:08:40,579 WARN
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: User 'user1' from
application tag does not have access to queue 'user1'. The placement is done
for user 'hive'
Checking the code, scheduler#checkAccess() bails out even before checking the
ACL permissions for that particular queue because the CSQueue is null.
{code:java}
public boolean checkAccess(UserGroupInformation callerUGI,
QueueACL acl, String queueName) {
CSQueue queue = getQueue(queueName);
if (queue == null) {
if (LOG.isDebugEnabled())
{ LOG.debug("ACL not found for queue access-type " + acl + " for queue " +
queueName); }
return false;*<-- the method returns false here.*
}
return queue.hasAccess(acl, callerUGI);
}
{code}
As this is an auto created queue, CSQueue may be null in this case. May be
scheduler#checkAccess() should have a logic to differentiate when CSQueue is
null and if queue mapping is involved and if so, check if the parent queue
exists and is a managed parent and if so, check if the parent queue has valid
ACL's instead of returning false ?
Thanks
was:
Recently, one of our customers created dynamic queues based on placement rules
in CDP Private Cloud Base 71.2 to run their Hive on Tez queries but the job
failed because of not submitting to the appropriate queue.
Analyzing the Resource Manager log, we could see that the queue creation fails
because ACL submit application check couldn't succeed.
We tried setting acl_submit_applications to '*' for managed parent queues. For
static queues, this worked but failed for dynamic queues. Also tried setting
the below property but it didn't help either.
yarn.scheduler.capacity.root.parent-queue-name.leaf-queue-template.acl_submit_applications=*.
RM error log shows the following :
2020-09-18 01:08:40,579 INFO
org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule:
Application application_1600399068816_0460 user user1 mapping [default] to
[queue1] override false
2020-09-18 01:08:40,579 WARN
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: User 'user1' from
application tag does not have access to queue 'user1'. The placement is done
for user 'hive'
Checking the code, scheduler#checkAccess() bails out even before checking the
ACL permissions for that particular queue because the CSQueue is null.
public boolean checkAccess(UserGroupInformation callerUGI,
QueueACL acl, String queueName) {
CSQueue queue = getQueue(queueName);
if (queue == null) {
if (LOG.isDebugEnabled())
{ LOG.debug("ACL not found for queue access-type " + acl + " for queue " +
queueName); }
return false;*<-- the method returns false here.*
}
return queue.hasAccess(acl, callerUGI);
}
As this is an auto created queue, CSQueue may be null in this case. May be
scheduler#checkAccess() should have a logic to differentiate when CSQueue is
null and if queue mapping is involved and if so, check if the parent queue
exists and is a managed parent and if so, check if the parent queue has valid
ACL's instead of returning false ?
Thanks
> Hive On Tez queries fails upon submission to dynamically created pools
> --
>
> Key: YARN-10458
> URL: https://issues.apache.org/jira/browse/YARN-10458
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Anand Srinivasan
>Priority: Major
>
> While using Dynamic Auto-Creation and Management of Leaf Queues, we could see
> that the queue creation fails because ACL submit application check couldn't
> succeed.
> We tried setting acl_submit_applications to '*' for managed parent queues.
> For static queues, this worked but failed for dynamic queues. Also tried
> setting the below property but it didn't help either.
> yarn.scheduler.capacity.root.parent-queue-name.leaf-queue-template.acl_submit_applications=*.
> RM error log shows the following :
>
