[
https://issues.apache.org/jira/browse/SPARK-21689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhoukang closed SPARK-21689.
----------------------------
Resolution: Won't Fix
> Spark submit will not get kerberos token token when hbase class not found
> -------------------------------------------------------------------------
>
> Key: SPARK-21689
> URL: https://issues.apache.org/jira/browse/SPARK-21689
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 2.1.0, 2.2.0
> Reporter: zhoukang
>
> When use yarn cluster mode,and we need scan hbase,there will be a case which
> can not work:
> If we put user jar on hdfs,when local classpath will has no hbase,which will
> let get hbase token failed.Then later when job submitted to yarn, it will
> failed since has no token to access hbase table.I mock three cases:
> 1：user jar is on classpath, and has hbase
> {code:java}
> 17/08/10 13:48:03 INFO security.HadoopFSDelegationTokenProvider: Renewal
> interval is 86400050 for token HDFS_DELEGATION_TOKEN
> 17/08/10 13:48:03 INFO security.HadoopDelegationTokenManager: Service hive
> 17/08/10 13:48:03 INFO security.HadoopDelegationTokenManager: Service hbase
> 17/08/10 13:48:05 INFO security.HBaseDelegationTokenProvider: Attempting to
> fetch HBase security token.
> {code}
> Logs showing we can get token normally.
> 2：user jar on hdfs
> {code:java}
> 17/08/10 13:43:58 WARN security.HBaseDelegationTokenProvider: Class
> org.apache.hadoop.hbase.HBaseConfiguration not found.
> 17/08/10 13:43:58 INFO security.HBaseDelegationTokenProvider: Failed to get
> token from service hbase
> java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.security.token.TokenUtil
> at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.spark.deploy.security.HBaseDelegationTokenProvider.obtainDelegationTokens(HBaseDelegationTokenProvider.scala:41)
> at
> org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:112)
> at
> org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:109)
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
> {code}
> Logs showing we can get token failed with ClassNotFoundException.
> If we download user jar from remote first,then things will work correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
