[jira] [Commented] (YARN-2563) On secure clusters call to timeline server fails with authentication errors when running a job via oozie
[ https://issues.apache.org/jira/browse/YARN-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140498#comment-14140498 ] Hudson commented on YARN-2563: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1901 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1901/]) YARN-2563. Fixed YarnClient to call getTimeLineDelegationToken only if the Token is not present. Contributed by Zhijie Shen (jianhe: rev eb92cc67dfaa51212fc5315b8db99effd046a154) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * hadoop-yarn-project/CHANGES.txt On secure clusters call to timeline server fails with authentication errors when running a job via oozie Key: YARN-2563 URL: https://issues.apache.org/jira/browse/YARN-2563 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 2.6.0 Reporter: Arpit Gupta Assignee: Zhijie Shen Priority: Blocker Fix For: 2.6.0 Attachments: YARN-2563.1.patch, YARN-2563.2.patch During our nightlies on a secure cluster we have seen oozie jobs fail with authentication error to the time line server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2563) On secure clusters call to timeline server fails with authentication errors when running a job via oozie
[ https://issues.apache.org/jira/browse/YARN-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140552#comment-14140552 ] Hudson commented on YARN-2563: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1876 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1876/]) YARN-2563. Fixed YarnClient to call getTimeLineDelegationToken only if the Token is not present. Contributed by Zhijie Shen (jianhe: rev eb92cc67dfaa51212fc5315b8db99effd046a154) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java On secure clusters call to timeline server fails with authentication errors when running a job via oozie Key: YARN-2563 URL: https://issues.apache.org/jira/browse/YARN-2563 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 2.6.0 Reporter: Arpit Gupta Assignee: Zhijie Shen Priority: Blocker Fix For: 2.6.0 Attachments: YARN-2563.1.patch, YARN-2563.2.patch During our nightlies on a secure cluster we have seen oozie jobs fail with authentication error to the time line server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2563) On secure clusters call to timeline server fails with authentication errors when running a job via oozie
[
https://issues.apache.org/jira/browse/YARN-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139376#comment-14139376
]
Jian He commented on YARN-2563:
---
not related to this bug, I think for the following we can just check the kind
name for simplicity.
{code}
TokenIdentifier tokenIdentifier = token.decodeIdentifier();
if (tokenIdentifier instanceof TimelineDelegationTokenIdentifier) {
return;
}
{code}
On secure clusters call to timeline server fails with authentication errors
when running a job via oozie
Key: YARN-2563
URL: https://issues.apache.org/jira/browse/YARN-2563
Project: Hadoop YARN
Issue Type: Bug
Components: timelineserver
Affects Versions: 2.6.0
Reporter: Arpit Gupta
Assignee: Zhijie Shen
Priority: Blocker
Attachments: YARN-2563.1.patch
During our nightlies on a secure cluster we have seen oozie jobs fail with
authentication error to the time line server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2563) On secure clusters call to timeline server fails with authentication errors when running a job via oozie
[
https://issues.apache.org/jira/browse/YARN-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139440#comment-14139440
]
Hadoop QA commented on YARN-2563:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12669779/YARN-2563.2.patch
against trunk revision 570b8b4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5025//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5025//console
This message is automatically generated.
On secure clusters call to timeline server fails with authentication errors
when running a job via oozie
Key: YARN-2563
URL: https://issues.apache.org/jira/browse/YARN-2563
Project: Hadoop YARN
Issue Type: Bug
Components: timelineserver
Affects Versions: 2.6.0
Reporter: Arpit Gupta
Assignee: Zhijie Shen
Priority: Blocker
Attachments: YARN-2563.1.patch, YARN-2563.2.patch
During our nightlies on a secure cluster we have seen oozie jobs fail with
authentication error to the time line server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2563) On secure clusters call to timeline server fails with authentication errors when running a job via oozie
[ https://issues.apache.org/jira/browse/YARN-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139542#comment-14139542 ] Jian He commented on YARN-2563: --- looks good, +1 On secure clusters call to timeline server fails with authentication errors when running a job via oozie Key: YARN-2563 URL: https://issues.apache.org/jira/browse/YARN-2563 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 2.6.0 Reporter: Arpit Gupta Assignee: Zhijie Shen Priority: Blocker Attachments: YARN-2563.1.patch, YARN-2563.2.patch During our nightlies on a secure cluster we have seen oozie jobs fail with authentication error to the time line server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2563) On secure clusters call to timeline server fails with authentication errors when running a job via oozie
[ https://issues.apache.org/jira/browse/YARN-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14138107#comment-14138107 ] Zhijie Shen commented on YARN-2563: --- When submitting an app in a secure mode, YarnClient will automatically obtain a timeline DT from the timeline server. This communication needs to pass Kerberos authentication. It works at the client side, which has Kerberos setup. In a container (either of AM or a specific task), the process doesn't do Kerberos login, such that it is not able to pass Kerberos authentication to get the timeline DT. In this scenario, Oozie is starting a MR job inside the MR mapper container, such that it fails to pass Kerberos authentication enforced by the timeline server. However, the expected behavior is that YarnClient only grab a timeline DT when it is not found when submitting a app, and the DT will be put into the credentials of ContainerLaunchContext, and passed to AM and the remaining MR tasks' containers. Hence when Oozie wants to launch to a RM job from there, it should already have the DT, and don't need to invoke getTimelineDelegationToken method. It seems that YarnClientImpl.addTimelineDelegationToken has a bug. No matter the DT is already in the credentials or not, YarnClientImpl will always grab one, but only put it into the credentials when the DT is not there. The right behavior should be: when the DT is already in credentials, we even shouldn't invoke getTimelineDelegationToken. I'll create a patch to fix the bug. On secure clusters call to timeline server fails with authentication errors when running a job via oozie Key: YARN-2563 URL: https://issues.apache.org/jira/browse/YARN-2563 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 2.6.0 Reporter: Arpit Gupta Assignee: Zhijie Shen Priority: Blocker During our nightlies on a secure cluster we have seen oozie jobs fail with authentication error to the time line server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2563) On secure clusters call to timeline server fails with authentication errors when running a job via oozie
[
https://issues.apache.org/jira/browse/YARN-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14138262#comment-14138262
]
Hadoop QA commented on YARN-2563:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12669557/YARN-2563.1.patch
against trunk revision 123f20d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5009//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5009//console
This message is automatically generated.
On secure clusters call to timeline server fails with authentication errors
when running a job via oozie
Key: YARN-2563
URL: https://issues.apache.org/jira/browse/YARN-2563
Project: Hadoop YARN
Issue Type: Bug
Components: timelineserver
Affects Versions: 2.6.0
Reporter: Arpit Gupta
Assignee: Zhijie Shen
Priority: Blocker
Attachments: YARN-2563.1.patch
During our nightlies on a secure cluster we have seen oozie jobs fail with
authentication error to the time line server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2563) On secure clusters call to timeline server fails with authentication errors when running a job via oozie
[
https://issues.apache.org/jira/browse/YARN-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136486#comment-14136486
]
Arpit Gupta commented on YARN-2563:
---
Here is the stack trace
{code}
ozie-oozi-W@wc] Launcher ERROR, reason: Main class
[org.apache.oozie.action.hadoop.MapReduceMain], main() threw exception,
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
2014-09-13 00:00:18,595 WARN MapReduceActionExecutor:546 - USER[hrt_qa]
GROUP[-] TOKEN[] APP[wordcount-wf] JOB[001-140912235159138-oozie-oozi-W]
ACTION[001-140912235159138-oozie-oozi-W@wc] Launcher exception:
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
java.io.IOException:
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
at
org.apache.hadoop.yarn.client.api.impl.TimelineAuthenticator.getDelegationToken(TimelineAuthenticator.java:135)
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.getDelegationToken(TimelineClientImpl.java:181)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.addTimelineDelegationToken(YarnClientImpl.java:275)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:221)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:282)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:289)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:437)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1294)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1291)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1291)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at
org.apache.oozie.action.hadoop.MapReduceMain.submitJob(MapReduceMain.java:108)
at
org.apache.oozie.action.hadoop.MapReduceMain.run(MapReduceMain.java:66)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39)
at
org.apache.oozie.action.hadoop.MapReduceMain.main(MapReduceMain.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by:
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
at
org.apache.hadoop.yarn.client.api.impl.TimelineAuthenticator.authenticate(TimelineAuthenticator.java:101)
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
at
org.apache.hadoop.yarn.client.api.impl.TimelineAuthenticator.getDelegationToken(TimelineAuthenticator.java:121)
