[
https://issues.apache.org/jira/browse/YARN-7787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340186#comment-16340186
]
Eric Yang commented on YARN-7787:
---------------------------------
YARN Service code contains one implementation of ApplicationMaster code that
extends
org.apache.hadoop.service.AbstractService. AM's responsibility is to report
service status, and other application logic. Hadoop RPC setup by
ApplicationMaster must follow basic Hadoop security practice. HADOOP-9698
added logic to make sure saslRPCClient verifies server side credential against
list of configuration defined principal names. The goal is to prevents men in
middle attack or replay attack. This is hard coded into Hadoop security design
when service are statically deploy on cluster of nodes.
Therefore, user must use server principal in Yarn Service to launch YARN
service:
{code:java}
"kerberos_principal" : {
"principal_name" : "hbase/[email protected]",
"keytab" : "file:///etc/security/keytabs/hbase.service.keytab"
},{code}
This ticket is to discuss whether there is any wiggle room to relax security
and allow end user principal to be used for starting service.
ApplicationMaster can run on any node in YARN cluster. This security check
seems cumbersome to generate a keytab that contains the proper server
principals for ApplicationMaster. In large scale cluster, using server
principal is definitely preferred to prevent men-in-middle attack even within
trusted security perimeter. This request can have profound impact to Hadoop
security design for sasl rpc client and worthy of discussion. The alternative
is to reimplement AM not base on Hadoop RPC, and new implementation needs to
solve men-in-middle attack in other shape or forms. It seems like a lot
disadvantages to enable end user principal to run ApplicationMaster. Thoughts?
> Yarn service can not be launched with User Principal
> ----------------------------------------------------
>
> Key: YARN-7787
> URL: https://issues.apache.org/jira/browse/YARN-7787
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
> Reporter: Yesha Vora
> Priority: Critical
>
> Steps:
> 1) update httpd.json by adding below block.
> {code:java}
> "kerberos_principal" : {
> "principal_name" : "[email protected]",
> "keytab" : "file:///home/hrt_qa/hadoopqa/keytabs/hrt_qa.headless.keytab"
> }{code}
> 2) Launch http example as hrt_qa user
> {code:java}
> 2018-01-19 22:00:37,238|INFO|MainThread|machine.py:150 -
> run()||GUID=6b0714d0-1377-43ee-8959-9ae380e1486c|RUNNING:
> /usr/hdp/current/hadoop-yarn-client/bin/yarn app -launch httpd-hrt-qa httpd
> 2018-01-19 22:00:37,295|INFO|WARNING: YARN_LOG_DIR has been replaced by
> HADOOP_LOG_DIR. Using value of YARN_LOG_DIR.
> 2018-01-19 22:00:37,295|INFO|WARNING: YARN_LOGFILE has been replaced by
> HADOOP_LOGFILE. Using value of YARN_LOGFILE.
> 2018-01-19 22:00:37,295|INFO|WARNING: YARN_PID_DIR has been replaced by
> HADOOP_PID_DIR. Using value of YARN_PID_DIR.
> 2018-01-19 22:00:37,296|INFO|WARNING: YARN_OPTS has been replaced by
> HADOOP_OPTS. Using value of YARN_OPTS.
> 2018-01-19 22:00:38,173|INFO|18/01/19 22:00:38 WARN util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 2018-01-19 22:00:39,530|INFO|18/01/19 22:00:39 WARN
> shortcircuit.DomainSocketFactory: The short-circuit local reads feature
> cannot be used because libhadoop cannot be loaded.
> 2018-01-19 22:00:39,545|INFO|18/01/19 22:00:39 INFO client.ServiceClient:
> Loading service definition from local FS:
> /usr/hdp/3.0.0.0-xx/hadoop-yarn/yarn-service-examples/httpd/httpd.json
> 2018-01-19 22:00:40,186|INFO|18/01/19 22:00:40 INFO
> client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
> 2018-01-19 22:00:40,492|INFO|18/01/19 22:00:40 INFO client.ServiceClient:
> Persisted service httpd-hrt-qa at
> hdfs://mycluster/user/hrt_qa/.yarn/services/httpd-hrt-qa/httpd-hrt-qa.json
> 2018-01-19 22:00:40,589|INFO|18/01/19 22:00:40 INFO conf.Configuration: found
> resource resource-types.xml at
> file:/etc/hadoop/3.0.0.0-xx/0/resource-types.xml
> 2018-01-19 22:00:40,719|INFO|18/01/19 22:00:40 INFO client.ServiceClient:
> Uploading all dependency jars to HDFS. For faster submission of apps,
> pre-upload dependency jars to HDFS using command: yarn app -enableFastLaunch
> 2018-01-19 22:00:48,253|INFO|18/01/19 22:00:48 INFO hdfs.DFSClient: Created
> token for hrt_qa: HDFS_DELEGATION_TOKEN [email protected],
> renewer=yarn, realUser=, issueDate=1516399248244, maxDate=1517004048244,
> sequenceNumber=4, masterKeyId=4 on ha-hdfs:mycluster
> 2018-01-19 22:00:49,463|INFO|18/01/19 22:00:49 INFO impl.YarnClientImpl:
> Submitted application application_1516398459631_0001{code}
> 3) Run "yarn application -status <appname>"
> {code:java}
> 2018-01-19 22:01:05,570|INFO|RUNNING:
> /usr/hdp/current/hadoop-yarn-client/bin/yarn application -status httpd-hrt-qa
> 2018-01-19 22:01:05,626|INFO|WARNING: YARN_LOG_DIR has been replaced by
> HADOOP_LOG_DIR. Using value of YARN_LOG_DIR.
> 2018-01-19 22:01:05,626|INFO|WARNING: YARN_LOGFILE has been replaced by
> HADOOP_LOGFILE. Using value of YARN_LOGFILE.
> 2018-01-19 22:01:05,626|INFO|WARNING: YARN_PID_DIR has been replaced by
> HADOOP_PID_DIR. Using value of YARN_PID_DIR.
> 2018-01-19 22:01:05,626|INFO|WARNING: YARN_OPTS has been replaced by
> HADOOP_OPTS. Using value of YARN_OPTS.
> 2018-01-19 22:01:06,529|INFO|18/01/19 22:01:06 WARN util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> 2018-01-19 22:01:07,851|INFO|18/01/19 22:01:07 WARN
> shortcircuit.DomainSocketFactory: The short-circuit local reads feature
> cannot be used because libhadoop cannot be loaded.
> 2018-01-19 22:01:08,003|INFO|18/01/19 22:01:08 INFO utils.ServiceApiUtil:
> Loading service definition from
> hdfs://mycluster/user/hrt_qa/.yarn/services/httpd-hrt-qa/httpd-hrt-qa.json
> 2018-01-19 22:01:08,563|INFO|18/01/19 22:01:08 INFO
> client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
> 2018-01-19 22:01:08,787|INFO|Exception in thread "main" java.io.IOException:
> Failed on local exception: java.io.IOException: Couldn't set up IO streams:
> java.lang.IllegalArgumentException: Kerberos principal name does NOT have the
> expected hostname part: [email protected]; Host Details : local host is:
> “host1/xx.xx.xx.xx"; destination host is: “host1”:40318;
> 2018-01-19 22:01:08,788|INFO|at
> org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:808)
> 2018-01-19 22:01:08,788|INFO|at
> org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1495)
> 2018-01-19 22:01:08,788|INFO|at
> org.apache.hadoop.ipc.Client.call(Client.java:1437)
> 2018-01-19 22:01:08,788|INFO|at
> org.apache.hadoop.ipc.Client.call(Client.java:1347)
> 2018-01-19 22:01:08,789|INFO|at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> 2018-01-19 22:01:08,789|INFO|at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> 2018-01-19 22:01:08,789|INFO|at com.sun.proxy.$Proxy40.getStatus(Unknown
> Source)
> 2018-01-19 22:01:08,789|INFO|at
> org.apache.hadoop.yarn.service.impl.pb.client.ClientAMProtocolPBClientImpl.getStatus(ClientAMProtocolPBClientImpl.java:68)
> 2018-01-19 22:01:08,789|INFO|at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2018-01-19 22:01:08,789|INFO|at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2018-01-19 22:01:08,790|INFO|at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2018-01-19 22:01:08,790|INFO|at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2018-01-19 22:01:08,790|INFO|at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> 2018-01-19 22:01:08,790|INFO|at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> 2018-01-19 22:01:08,790|INFO|at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> 2018-01-19 22:01:08,790|INFO|at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> 2018-01-19 22:01:08,791|INFO|at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> 2018-01-19 22:01:08,791|INFO|at com.sun.proxy.$Proxy41.getStatus(Unknown
> Source)
> 2018-01-19 22:01:08,791|INFO|at
> org.apache.hadoop.yarn.service.client.ServiceClient.getStatus(ServiceClient.java:958)
> 2018-01-19 22:01:08,791|INFO|at
> org.apache.hadoop.yarn.service.client.ServiceClient.getStatusString(ServiceClient.java:910)
> 2018-01-19 22:01:08,791|INFO|at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:316)
> 2018-01-19 22:01:08,791|INFO|at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> 2018-01-19 22:01:08,792|INFO|at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> 2018-01-19 22:01:08,792|INFO|at
> org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:111)
> 2018-01-19 22:01:08,792|INFO|Caused by: java.io.IOException: Couldn't set up
> IO streams: java.lang.IllegalArgumentException: Kerberos principal name does
> NOT have the expected hostname part: [email protected]
> 2018-01-19 22:01:08,792|INFO|at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:860)
> 2018-01-19 22:01:08,792|INFO|at
> org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:409)
> 2018-01-19 22:01:08,792|INFO|at
> org.apache.hadoop.ipc.Client.getConnection(Client.java:1552)
> 2018-01-19 22:01:08,793|INFO|at
> org.apache.hadoop.ipc.Client.call(Client.java:1383)
> 2018-01-19 22:01:08,793|INFO|... 21 more
> 2018-01-19 22:01:08,793|INFO|Caused by: java.lang.IllegalArgumentException:
> Kerberos principal name does NOT have the expected hostname part:
> [email protected]
> 2018-01-19 22:01:08,793|INFO|at
> org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:332)
> 2018-01-19 22:01:08,793|INFO|at
> org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:234)
> 2018-01-19 22:01:08,793|INFO|at
> org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:160)
> 2018-01-19 22:01:08,794|INFO|at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
> 2018-01-19 22:01:08,794|INFO|at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:613)
> 2018-01-19 22:01:08,794|INFO|at
> org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:409)
> 2018-01-19 22:01:08,794|INFO|at
> org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:798)
> 2018-01-19 22:01:08,795|INFO|at
> org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:794)
> 2018-01-19 22:01:08,795|INFO|at
> java.security.AccessController.doPrivileged(Native Method)
> 2018-01-19 22:01:08,795|INFO|at
> javax.security.auth.Subject.doAs(Subject.java:422)
> 2018-01-19 22:01:08,795|INFO|at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> 2018-01-19 22:01:08,795|INFO|at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:794)
> 2018-01-19 22:01:08,795|INFO|... 24 more
> {code}
> yarn application -status should not fail with
> "java.lang.IllegalArgumentException: Kerberos principal name does NOT have
> the expected hostname part: [email protected]".
> It should accept the principal name without hostname.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
