zhoukang created SPARK-21689:
--------------------------------
Summary: Spark submit will not get kerberos token token when hbase
class not found
Key: SPARK-21689
URL: https://issues.apache.org/jira/browse/SPARK-21689
Project: Spark
Issue Type: Improvement
Components: YARN
Affects Versions: 2.2.0, 2.1.0
Reporter: zhoukang
When use yarn cluster mode,and we need scan hbase,there will be a case which
can not work:
If we put user jar on hdfs,when local classpath will has no hbase,which will
let get hbase token failed.Then later when job submitted to yarn, it will
failed since has no token to access hbase table.I mock three cases:
1：user jar is on classpath, and has hbase
{code:java}
17/08/10 13:48:03 INFO security.HadoopFSDelegationTokenProvider: Renewal
interval is 86400050 for token HDFS_DELEGATION_TOKEN
17/08/10 13:48:03 INFO security.HadoopDelegationTokenManager: Service hive
17/08/10 13:48:03 INFO security.HadoopDelegationTokenManager: Service hbase
17/08/10 13:48:05 INFO security.HBaseDelegationTokenProvider: Attempting to
fetch HBase security token.
{code}
2：user jar on hdfs
{code:java}
17/08/10 13:43:58 WARN security.HBaseDelegationTokenProvider: Class
org.apache.hadoop.hbase.HBaseConfiguration not found.
17/08/10 13:43:58 INFO security.HBaseDelegationTokenProvider: Failed to get
token from service hbase
java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.security.token.TokenUtil
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.spark.deploy.security.HBaseDelegationTokenProvider.obtainDelegationTokens(HBaseDelegationTokenProvider.scala:41)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:112)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:109)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
{code}
If we download user jar from remote first,then things will work correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
