[ 
https://issues.apache.org/jira/browse/SPARK-21689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121124#comment-16121124
 ] 

zhoukang commented on SPARK-21689:
----------------------------------

https://github.com/apache/spark/pull/18901 i created a pr for this issue.And i 
wonder why the pr did not related to this issue automatically?

> Spark submit will not get kerberos token token when hbase class not found
> -------------------------------------------------------------------------
>
>                 Key: SPARK-21689
>                 URL: https://issues.apache.org/jira/browse/SPARK-21689
>             Project: Spark
>          Issue Type: Improvement
>          Components: YARN
>    Affects Versions: 2.1.0, 2.2.0
>            Reporter: zhoukang
>
> When use yarn cluster mode,and we need scan hbase,there will be a case which 
> can not work:
> If we put user jar on hdfs,when local classpath will has no hbase,which will 
> let get hbase token failed.Then later when job submitted to yarn, it will 
> failed since has no token to access hbase table.I mock three cases:
> 1:user jar is on classpath, and has hbase
> {code:java}
> 17/08/10 13:48:03 INFO security.HadoopFSDelegationTokenProvider: Renewal 
> interval is 86400050 for token HDFS_DELEGATION_TOKEN
> 17/08/10 13:48:03 INFO security.HadoopDelegationTokenManager: Service hive
> 17/08/10 13:48:03 INFO security.HadoopDelegationTokenManager: Service hbase
> 17/08/10 13:48:05 INFO security.HBaseDelegationTokenProvider: Attempting to 
> fetch HBase security token.
> {code}
> Logs showing we can get token normally.
> 2:user jar on hdfs
> {code:java}
> 17/08/10 13:43:58 WARN security.HBaseDelegationTokenProvider: Class 
> org.apache.hadoop.hbase.HBaseConfiguration not found.
> 17/08/10 13:43:58 INFO security.HBaseDelegationTokenProvider: Failed to get 
> token from service hbase
> java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.security.token.TokenUtil
>       at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
>       at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>       at java.security.AccessController.doPrivileged(Native Method)
>       at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
>       at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>       at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>       at 
> org.apache.spark.deploy.security.HBaseDelegationTokenProvider.obtainDelegationTokens(HBaseDelegationTokenProvider.scala:41)
>       at 
> org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:112)
>       at 
> org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:109)
>       at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>       at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
> {code}
> Logs showing we can get token failed with ClassNotFoundException.
> If we download user jar from remote first,then things will work correctly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to