[
https://issues.apache.org/jira/browse/SPARK-22290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saisai Shao resolved SPARK-22290.
---------------------------------
Resolution: Fixed
Fix Version/s: 2.3.0
Issue resolved by pull request 19509
[https://github.com/apache/spark/pull/19509]
> Starting second context in same JVM fails to get new Hive delegation token
> --------------------------------------------------------------------------
>
> Key: SPARK-22290
> URL: https://issues.apache.org/jira/browse/SPARK-22290
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.3.0
> Reporter: Marcelo Vanzin
> Fix For: 2.3.0
>
>
> Consider the following pyspark script:
> {code}
> sc = SparkContext()
> // do stuff
> sc.stop()
> // do some other stuff
> sc = SparkContext()
> {code}
> That code didn't use to work at all in 2.2 (failure to create the second
> context), but makes more progress in 2.3. But it fails to create new Hive
> delegation tokens; you see this error in the output:
> {noformat}
> 17/10/16 16:26:50 INFO security.HadoopFSDelegationTokenProvider: getting
> token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1714191595_19,
> ugi=blah(auth:KERBEROS)]]
> 17/10/16 16:26:50 INFO hive.metastore: Trying to connect to metastore with
> URI blah
> 17/10/16 16:26:50 INFO hive.metastore: Connected to metastore.
> 17/10/16 16:26:50 ERROR metadata.Hive: MetaException(message:Delegation Token
> can be issued only with kerberos authentication. Current
> AuthenticationMethod: TOKEN)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_delegation_token_result$get_delegation_token_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_delegation_token_result$get_delegation_token_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_delegation_token_result.read(ThriftHiveMetastore
> {noformat}
> The error is printed in the logs but it doesn't cause the app to fail (which
> might be considered wrong).
> The effect is that when that old delegation token expires the new app will
> fail.
> But the real issue here is that Spark shouldn't be mixing delegation tokens
> from different apps. It should try harder to isolate a set of delegation
> tokens to a single app submission.
> And, in the case of Hive, there are many situations where a delegation token
> isn't needed at all.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
