HeartSaVioR commented on a change in pull request #23605:
[SPARK-26650][CORE]Yarn Client throws 'ClassNotFoundException:
org.apache.hadoop.hbase.HBaseConfiguration'
URL: https://github.com/apache/spark/pull/23605#discussion_r249576591
##########
File path:
core/src/main/scala/org/apache/spark/deploy/security/HBaseDelegationTokenProvider.scala
##########
@@ -70,6 +70,10 @@ private[security] class HBaseDelegationTokenProvider
getMethod("create", classOf[Configuration])
confCreate.invoke(null, conf).asInstanceOf[Configuration]
} catch {
+ case e: ClassNotFoundException =>
+ logWarning(s"You are attempting to use the
${getClass.getCanonicalName}, but your" +
Review comment:
Unlike Hadoop and Hive, Spark will not be built with HBase libs. It will
only work when end users link the HBase libraries or contain HBase libraries in
their app jar.
So the message doesn't look correct, though I think `Fail to invoke
HBaseConfiguration` is more misleading.
Actually the root issue is that HBaseDelegationToken is activated even end
users don't use HBase in their apps, which #23499 is trying to avoid with
service loader.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
