Weiqing Yang created AMBARI-21028:
-------------------------------------
Summary: The credential cache for livy is messed up
Key: AMBARI-21028
URL: https://issues.apache.org/jira/browse/AMBARI-21028
Project: Ambari
Issue Type: Bug
Components: ambari-server
Affects Versions: trunk
Reporter: Weiqing Yang
Assignee: Weiqing Yang
This issue was reported by [~kbadani].
Steps to reproduce this issue:
* Kdestroy and kinit as 'livy' user
* Do spark-submit with --proxy-user as 'hrt_1'
* In the console output, you can see that 'ambari-qa' is trying to impersonate
as 'hrt_1' and its failing
* Cancel the running job and do klist again - it will show credentials for
'ambari-qa' user and not the 'livy' user with which it was kinited
{code:java}
[livy@ctr-e133-1493418528701-6489-01-000003 spark]$ kinit -kt
/etc/security/keytabs/livy.service.keytab
livy/ctr-e133-1493418528701-6489-01-000003.hwx.site
[livy@ctr-e133-1493418528701-6489-01-000003 spark]$ klist
Ticket cache: FILE:/tmp/krb5cc_1808
Default principal:
livy/[email protected]
Valid starting Expires Service principal
05/02/2017 23:52:14 05/03/2017 23:52:14 krbtgt/[email protected]
[livy@ctr-e133-1493418528701-6489-01-000003 spark]$ spark-submit --class
org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors 3
--driver-memory 512m --executor-memory 512m --proxy-user hrt_1 --executor-cores
1
/usr/hdp/current/spark-client/lib/spark-examples-1.6.3.2.6.1.0-45-hadoop2.7.3.2.6.1.0-45.jar
10
Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set
Spark1 will be picked by default
17/05/02 23:53:10 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
17/05/02 23:53:12 INFO AHSProxy: Connecting to Application History server at
ctr-e133-1493418528701-6489-01-000004.hwx.site/172.27.22.136:10200
17/05/02 23:53:12 INFO RequestHedgingRMFailoverProxyProvider: Looking for the
active RM in [rm1, rm2]...
17/05/02 23:53:12 WARN RequestHedgingRMFailoverProxyProvider: Invocation
returned exception on [rm1] :
org.apache.hadoop.security.authorize.AuthorizationException: User:
[email protected] is not allowed to impersonate hrt_1, so propagating back
to caller.
17/05/02 23:53:12 INFO RequestHedgingRMFailoverProxyProvider: Connection lost
with rm1, trying to fail over.
17/05/02 23:53:12 INFO RequestHedgingRMFailoverProxyProvider: Looking for the
active RM in [rm1, rm2]...
17/05/02 23:53:12 WARN RequestHedgingRMFailoverProxyProvider: Invocation
returned exception on [rm1] :
org.apache.hadoop.security.authorize.AuthorizationException: User:
[email protected] is not allowed to impersonate hrt_1, so propagating back
to caller.
17/05/02 23:53:12 INFO RetryInvocationHandler:
org.apache.hadoop.security.authorize.AuthorizationException: User:
[email protected] is not allowed to impersonate hrt_1, while invoking
$Proxy9.getClusterMetrics over Failover proxy for [rm1, rm2] after 1 failover
attempts. Trying to failover after sleeping for 10442ms.
[livy@ctr-e133-1493418528701-6489-01-000003 spark]$ klist
Ticket cache: FILE:/tmp/krb5cc_1808
Default principal: [email protected]
Valid starting Expires Service principal
05/02/2017 23:53:03 05/03/2017 23:53:03 krbtgt/[email protected]
05/02/2017 23:53:03 05/03/2017 23:53:03
HTTP/[email protected]
{code}
Root cause is:
The livy smoke test launched by Ambari is run as livy user, but kinits as
ambari-qa, and therefore messes up the credential cache for livy.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
