[jira] [Commented] (YARN-2704) Localization and log-aggregation will fail if hdfs delegation token expired after token-max-life-time
[ https://issues.apache.org/jira/browse/YARN-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363320#comment-14363320 ] Bo Wang commented on YARN-2704: --- Why not guard the call of requestNewHdfsDelegationToken with hasProxyUserPrivileges in handleAppSubmitEvent in DelegationTokenRenewer.java? RM I see you do guard the call with hasProxyUserPrivileges in requestNewHdfsDelegationTokenIfNeeded. With this patch, applications, whose client doesn't pass hdfs delegation tokens to RM, will fail to start with diagnostics $yarn_user is not allowed to impersonate $app_user. Localization and log-aggregation will fail if hdfs delegation token expired after token-max-life-time -- Key: YARN-2704 URL: https://issues.apache.org/jira/browse/YARN-2704 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Priority: Critical Fix For: 2.6.0 Attachments: YARN-2704.1.patch, YARN-2704.2.patch, YARN-2704.2.patch, YARN-2704.3.patch, YARN-2704.4.patch In secure mode, YARN requires the hdfs-delegation token to do localization and log aggregation on behalf of the user. But the hdfs delegation token will eventually expire after max-token-life-time. So, localization and log aggregation will fail after the token expires. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2704) Localization and log-aggregation will fail if hdfs delegation token expired after token-max-life-time
[ https://issues.apache.org/jira/browse/YARN-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363770#comment-14363770 ] Bo Wang commented on YARN-2704: --- Thank you for your quick reply. You are correct. The problem is that I failed to add resource manager as a hadoop proxy user in core-site.xml. Localization and log-aggregation will fail if hdfs delegation token expired after token-max-life-time -- Key: YARN-2704 URL: https://issues.apache.org/jira/browse/YARN-2704 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Priority: Critical Fix For: 2.6.0 Attachments: YARN-2704.1.patch, YARN-2704.2.patch, YARN-2704.2.patch, YARN-2704.3.patch, YARN-2704.4.patch In secure mode, YARN requires the hdfs-delegation token to do localization and log aggregation on behalf of the user. But the hdfs delegation token will eventually expire after max-token-life-time. So, localization and log aggregation will fail after the token expires. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-53) Add protocol to YARN to support GetGroups
[
https://issues.apache.org/jira/browse/YARN-53?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bo Wang updated YARN-53:
Attachment: YARN-53-v5.patch
Added a testcase for GetGroups
Add protocol to YARN to support GetGroups
-
Key: YARN-53
URL: https://issues.apache.org/jira/browse/YARN-53
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Alejandro Abdelnur
Assignee: Bo Wang
Labels: patch
Fix For: 2.0.0-alpha
Attachments: MAPREDUCE-4268.patch, YARN-53.patch, YARN-53-v2.patch,
YARN-53-v3.patch, YARN-53-v4.patch, YARN-53-v5.patch
This is a regression from Hadoop1, as hadoop mrgroups fails with:
{code}
Exception in thread main java.lang.UnsupportedOperationException
at
org.apache.hadoop.mapred.tools.GetGroups.getProtocolAddress(GetGroups.java:50)
at
org.apache.hadoop.tools.GetGroupsBase.getUgmProtocol(GetGroupsBase.java:98)
at org.apache.hadoop.tools.GetGroupsBase.run(GetGroupsBase.java:71)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.mapred.tools.GetGroups.main(GetGroups.java:54)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-53) Add protocol to YARN to support GetGroups
[
https://issues.apache.org/jira/browse/YARN-53?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bo Wang updated YARN-53:
Affects Version/s: (was: 2.0.0-alpha)
Add protocol to YARN to support GetGroups
-
Key: YARN-53
URL: https://issues.apache.org/jira/browse/YARN-53
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Alejandro Abdelnur
Assignee: Bo Wang
Labels: patch
Fix For: 2.0.3-alpha
Attachments: MAPREDUCE-4268.patch, YARN-53.patch
This is a regression from Hadoop1, as hadoop mrgroups fails with:
{code}
Exception in thread main java.lang.UnsupportedOperationException
at
org.apache.hadoop.mapred.tools.GetGroups.getProtocolAddress(GetGroups.java:50)
at
org.apache.hadoop.tools.GetGroupsBase.getUgmProtocol(GetGroupsBase.java:98)
at org.apache.hadoop.tools.GetGroupsBase.run(GetGroupsBase.java:71)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.mapred.tools.GetGroups.main(GetGroups.java:54)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-53) Add protocol to YARN to support GetGroups
[
https://issues.apache.org/jira/browse/YARN-53?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bo Wang updated YARN-53:
Attachment: YARN-53.patch
Add protocol to YARN to support GetGroups
-
Key: YARN-53
URL: https://issues.apache.org/jira/browse/YARN-53
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.1.0-alpha
Reporter: Alejandro Abdelnur
Assignee: Bo Wang
Labels: patch
Fix For: 2.1.0-alpha
Attachments: MAPREDUCE-4268.patch, YARN-53.patch
This is a regression from Hadoop1, as hadoop mrgroups fails with:
{code}
Exception in thread main java.lang.UnsupportedOperationException
at
org.apache.hadoop.mapred.tools.GetGroups.getProtocolAddress(GetGroups.java:50)
at
org.apache.hadoop.tools.GetGroupsBase.getUgmProtocol(GetGroupsBase.java:98)
at org.apache.hadoop.tools.GetGroupsBase.run(GetGroupsBase.java:71)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.mapred.tools.GetGroups.main(GetGroups.java:54)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-53) Add protocol to YARN to support GetGroups
[
https://issues.apache.org/jira/browse/YARN-53?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450100#comment-13450100
]
Bo Wang commented on YARN-53:
-
Hi Vinod,
Thanks for looking at the patch. I'd like to clarify the two comments.
- To reflect YARN-1, do you mean move org.apache.hadoop.mapred.tools.GetGroups
to org.apache.hadoop.yarn.tools.GetGroups?
Currently there is no such package in hadoop-yarn-common yet, but I can
create one. Actually, bin/yarn rmadmin -getGroups doesn't call to
org.apache.hadoop.mapred.tools.GetGroups but
org.apache.hadoop.yarn.server.resourcemanager.tools.RMAdmin. GetGroups can be
left there for backward compatibility or removed directly.
- I am not sure what definitions in HDFS you referred to. Would you mind giving
it a bit more elaboration?
Thanks,
Bo
Add protocol to YARN to support GetGroups
-
Key: YARN-53
URL: https://issues.apache.org/jira/browse/YARN-53
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.1.0-alpha
Reporter: Alejandro Abdelnur
Assignee: Bo Wang
Labels: patch
Fix For: 2.1.0-alpha
Attachments: MAPREDUCE-4268.patch, YARN-53.patch
This is a regression from Hadoop1, as hadoop mrgroups fails with:
{code}
Exception in thread main java.lang.UnsupportedOperationException
at
org.apache.hadoop.mapred.tools.GetGroups.getProtocolAddress(GetGroups.java:50)
at
org.apache.hadoop.tools.GetGroupsBase.getUgmProtocol(GetGroupsBase.java:98)
at org.apache.hadoop.tools.GetGroupsBase.run(GetGroupsBase.java:71)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.mapred.tools.GetGroups.main(GetGroups.java:54)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-76) killApplication doesn't fully kill application master
Bo Wang created YARN-76: --- Summary: killApplication doesn't fully kill application master Key: YARN-76 URL: https://issues.apache.org/jira/browse/YARN-76 Project: Hadoop YARN Issue Type: Bug Environment: Failed on MacOS. OK on Linux Reporter: Bo Wang When client sends a ClientRMProtocol#killApplication to RM, the corresponding AM is supposed to be killed. However, on Mac OS, the AM is still alive (w/o any interruption). I figured out part of the reason after some debugging. NM starts a AM with command like /bin/bash -c /path/to/java SampleAM. This command is executed in a process (say with PID 0001), which starts another Java process (say with PID 0002). When NM kills the AM, it send SIGTERM and then SIGKILL to the bash process (PID 0001). In Linux, the death of the bash process (PID 0001) will trigger the kill of the Java process (PID 0002). However, in Mac OS, only the bash process is killed. The Java process is in the wild since then. Note: on Mac OS, DefaultContainerExecutor is used rather than LinuxContainerExecutor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-53) Add protocol to YARN to support GetGroups
[
https://issues.apache.org/jira/browse/YARN-53?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13444517#comment-13444517
]
Bo Wang commented on YARN-53:
-
Hi Arun, thanks for the feedback. I have a question that currently bin/yarn
rmadmin is a set of refresh commands. Would query commands (e.g. bin/mapred
groups) also be added there?
Add protocol to YARN to support GetGroups
-
Key: YARN-53
URL: https://issues.apache.org/jira/browse/YARN-53
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.1.0-alpha
Reporter: Alejandro Abdelnur
Assignee: Bo Wang
Attachments: MAPREDUCE-4268.patch
This is a regression from Hadoop1, as hadoop mrgroups fails with:
{code}
Exception in thread main java.lang.UnsupportedOperationException
at
org.apache.hadoop.mapred.tools.GetGroups.getProtocolAddress(GetGroups.java:50)
at
org.apache.hadoop.tools.GetGroupsBase.getUgmProtocol(GetGroupsBase.java:98)
at org.apache.hadoop.tools.GetGroupsBase.run(GetGroupsBase.java:71)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.mapred.tools.GetGroups.main(GetGroups.java:54)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
