[
https://issues.apache.org/jira/browse/SPARK-16298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358814#comment-15358814
]
Partha Pratim Ghosh commented on SPARK-16298:
---------------------------------------------
I have tried with spark-submit -
spark-submit --class connectivity.spark.java.Wordcount --executor-memory 12g
--principal <principal name> --keytab /app/<keytab file name>.keytab --master
yarn --deploy-mode cluster
/app/java/spark-connectivity/target/spark-connectivity-0.0.1-SNAPSHOT.jar
To be sure that it is using keytab I removed the kerberos cache file from /tmp
so that klist -f provides the following output -
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_515)
To be sure that the key is valid I checked the key with klist -k -t <keytab
file name>.keytab
Now, when I ran the above piece I received the following -
16/07/01 07:07:45 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 5958319
for pg18084 on ha-hdfs:FajitaDevelopment
Exception in thread "main" java.io.IOException:
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:794)
at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
...
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)
at
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
Am I missing something here?
Seems that even if I provide the principal and keytab spark-submit tries to
read from the default ticket cache in the unix session.
In that case, is there any requirement that the default unix keytab shall be
related to the principal and keytab I provide in some way?
> spark.yarn.principal not working
> --------------------------------
>
> Key: SPARK-16298
> URL: https://issues.apache.org/jira/browse/SPARK-16298
> Project: Spark
> Issue Type: Bug
> Reporter: Partha Pratim Ghosh
>
> I am opening a Spark configuration with spark.yarn.principal and
> spark.yarn.keytab. However, this is not authenticating the underlying HDFS
> with the same principal and keytab. Instead, seems it is picking up from
> ticket cache. Without this feature the spark.yarn.principal and
> spark.yarn.keytab doesn't seem to be logical.
> Sample code -
> SparkConf conf = new
> SparkConf().setMaster("yarn-client").setAppName("spark-test")
> .set("spark.repl.class.uri",
> classServerUri);
> conf.set("spark.yarn.principal", principal);
> conf.set("spark.yarn.keytab", keytab);
> conf.setSparkHome(sparkBasePath);
>
> if (execUri != null) {
> conf.set("spark.executor.uri", execUri);
> }
> conf.set("spark.executor.memory", "8g");
> conf.set("spark.scheduler.mode", "FAIR");
> SparkContext sparkContext = new
> SparkContext(conf);
> Please advise how this can be achieved.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
