[ https://issues.apache.org/jira/browse/AMBARI-21028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Weiqing Yang updated AMBARI-21028: ---------------------------------- Attachment: AMBARI-21028_v2.patch > The credential cache for livy is messed up > ------------------------------------------ > > Key: AMBARI-21028 > URL: https://issues.apache.org/jira/browse/AMBARI-21028 > Project: Ambari > Issue Type: Bug > Components: ambari-server > Affects Versions: trunk > Reporter: Weiqing Yang > Assignee: Weiqing Yang > Fix For: trunk > > Attachments: AMBARI-21028_v0.patch, AMBARI-21028_v1.patch, > AMBARI-21028_v2.patch > > > This issue was reported by [~kbadani]. > Steps to reproduce this issue: > * Kdestroy and kinit as 'livy' user > * Do spark-submit with --proxy-user as 'hrt_1' > * In the console output, you can see that 'ambari-qa' is trying to > impersonate as 'hrt_1' and its failing > * Cancel the running job and do klist again - it will show credentials for > 'ambari-qa' user and not the 'livy' user with which it was kinited > {code:java} > [livy@ctr-e133-1493418528701-6489-01-000003 spark]$ kinit -kt > /etc/security/keytabs/livy.service.keytab > livy/ctr-e133-1493418528701-6489-01-000003.hwx.site > [livy@ctr-e133-1493418528701-6489-01-000003 spark]$ klist > Ticket cache: FILE:/tmp/krb5cc_1808 > Default principal: > livy/ctr-e133-1493418528701-6489-01-000003.hwx.s...@example.com > Valid starting Expires Service principal > 05/02/2017 23:52:14 05/03/2017 23:52:14 krbtgt/example....@example.com > [livy@ctr-e133-1493418528701-6489-01-000003 spark]$ spark-submit --class > org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors 3 > --driver-memory 512m --executor-memory 512m --proxy-user hrt_1 > --executor-cores 1 > /usr/hdp/current/spark-client/lib/spark-examples-1.6.3.2.6.1.0-45-hadoop2.7.3.2.6.1.0-45.jar > 10 > Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set > Spark1 will be picked by default > 17/05/02 23:53:10 WARN NativeCodeLoader: Unable to load native-hadoop library > for your platform... using builtin-java classes where applicable > 17/05/02 23:53:12 INFO AHSProxy: Connecting to Application History server at > ctr-e133-1493418528701-6489-01-000004.hwx.site/172.27.22.136:10200 > 17/05/02 23:53:12 INFO RequestHedgingRMFailoverProxyProvider: Looking for the > active RM in [rm1, rm2]... > 17/05/02 23:53:12 WARN RequestHedgingRMFailoverProxyProvider: Invocation > returned exception on [rm1] : > org.apache.hadoop.security.authorize.AuthorizationException: User: > ambari...@example.com is not allowed to impersonate hrt_1, so propagating > back to caller. > 17/05/02 23:53:12 INFO RequestHedgingRMFailoverProxyProvider: Connection lost > with rm1, trying to fail over. > 17/05/02 23:53:12 INFO RequestHedgingRMFailoverProxyProvider: Looking for the > active RM in [rm1, rm2]... > 17/05/02 23:53:12 WARN RequestHedgingRMFailoverProxyProvider: Invocation > returned exception on [rm1] : > org.apache.hadoop.security.authorize.AuthorizationException: User: > ambari...@example.com is not allowed to impersonate hrt_1, so propagating > back to caller. > 17/05/02 23:53:12 INFO RetryInvocationHandler: > org.apache.hadoop.security.authorize.AuthorizationException: User: > ambari...@example.com is not allowed to impersonate hrt_1, while invoking > $Proxy9.getClusterMetrics over Failover proxy for [rm1, rm2] after 1 failover > attempts. Trying to failover after sleeping for 10442ms. > [livy@ctr-e133-1493418528701-6489-01-000003 spark]$ klist > Ticket cache: FILE:/tmp/krb5cc_1808 > Default principal: ambari...@example.com > Valid starting Expires Service principal > 05/02/2017 23:53:03 05/03/2017 23:53:03 krbtgt/example....@example.com > 05/02/2017 23:53:03 05/03/2017 23:53:03 > HTTP/ctr-e133-1493418528701-6489-01-000003.hwx.s...@example.com > {code} > Root cause is: > The livy smoke test launched by Ambari is run as livy user, but kinits as > ambari-qa, and therefore messes up the credential cache for livy. -- This message was sent by Atlassian JIRA (v6.3.15#6346)