Alexandre Linte created YARN-7618:
-------------------------------------
Summary: YARN REST APi - can't launch yarn job on Kerberised
Cluster
Key: YARN-7618
URL: https://issues.apache.org/jira/browse/YARN-7618
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.7.4
Environment: Hadoop 2.7.4 - Kerberized cluster
Reporter: Alexandre Linte
Priority: Critical
Hello,
I'm currently trying to launch a yarn job through an Hadoop kerberized cluster
following documentation (
https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
)
I'm doing these operation with an active kerberos keytab.
I begin to create my new-application
{code:title=curl new-app|borderStyle=solid}
curl --negotiate -u : -XPOST
http://uabigrm01.node.com:8088/ws/v1/cluster/apps/new-application
response :
{"application-id":"application_1507815642943_271826","maximum-resource-capability":{"memory":32768,"vCores":24}}
{code}
After that I submit my application :
{code:title=curl submit|borderStyle=solid}
curl --negotiate -u : -XPOST -H "Content-Type: application/json" --data
@"submit.json" http://uabigrm01.node.com:8088/ws/v1/cluster/apps
{code}
Content of submit.json file :
{code:title=submit.json|borderStyle=solid}
{
"application-id": "application_1507815642943_271826",
"application-name": "yarn-api-test-new",
"queue": "myqueue",
"am-container-spec": {
"commands": {
"command": "{{HADOOP_HOME}}/bin/yarn jar
/opt/application/Hadoop/current/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.4.jar
wordcount /user/mwxk0647/WORK/dataset-input
/user/mwxk0647/WORK/dataset-output-test-yarn"
},
"environment": {
"entry": [{
"key": "CLASSPATH",
"value":"{{CLASSPATH}}<CPS>./*<CPS>{{HADOOP_CONF_DIR}}<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/*<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/lib/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/lib/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/lib/*<CPS>./log4j.properties"
}]
}
},
"unmanaged-AM": false,
"max-app-attempts": 2,
"resource": {
"memory": 1024,
"vCores": 1
},
"application-type": "MAPREDUCE",
"keep-containers-across-application-attempts": false
}
{code}
I can see job on scheduler, he is submit but He failed due to a Kerberos
authentication error...
{code:title=tracelogs|borderStyle=solid}
User: mwxk0647
Name: yarn-api-test
Application Type: MAPREDUCE
Application Tags:
YarnApplicationState: FAILED
Queue: myqueue
FinalStatus Reported by AM: FAILED
Started: Wed Dec 06 14:45:56 +0100 2017
Elapsed: 10sec
Tracking URL: History
Diagnostics:
Application application_1507815642943_424552 failed 2 times due to AM Container
for appattempt_1507815642943_424552_000002 exited with exitCode: 255
For more detailed output, check application tracking
page:http://uabigrm01.node.com:8188/applicationhistory/app/application_1507815642943_424552Then,
click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1507815642943_424552_02_000001
Exit code: 255
Exception message: java.io.IOException: Failed on local exception:
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)]; Host Details : local host is:
"uabigdata69.node.com/10.77.64.69"; destination host is:
"uabigname02.node.com":8020;
{code}
So kerberos is OK for submit app, but not for launch job.
For the moment I make application works making manually my kinit on the
datanode :
{code:borderStyle=solid}
“command”: "echo <MYPASSWORD> | kinit mwxk0647 && {{HADOOP_HOME}}/bin/yarn
jar...
{code}
But It's really ugly... And on scheduler, it display a first job which fail as
before, but it launch the wordcount job, which is very strange..
How can i make YARN REST API work properly on a Kerberized Environment ?
Best Regards.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
