I'm trying to use Phoenix on Spark, and can't get around this error:
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
at
org.apache.phoenix.spark.PhoenixRDD.getPhoenixConfiguration(PhoenixRDD.scala:82)
DETAILS:
1. I'm running HDP 2.4.0.0-169
2. Using phoenix-sqlline, I can access Phoenix perfectly
3. Using hbase shell, I can access HBase perfectly
4. I added the following lines to /etc/spark/conf/spark-defaults.conf
spark.driver.extraClassPath
/usr/hdp/current/phoenix-client/lib/phoenix-spark-4.4.0.2.4.0.0-169.jar
spark.executor.extraClassPath
/usr/hdp/current/phoenix-client/lib/phoenix-spark-4.4.0.2.4.0.0-169.jar
5. Steps to reproduce the error:
# spark-shell
...
scala> import org.apache.phoenix.spark._
import org.apache.phoenix.spark._
scala> sqlContext.load("org.apache.phoenix.spark", Map("table" ->
"EMAIL_ENRON", "zkUrl" -> "localhost:2181"))
warning: there were 1 deprecation warning(s); re-run with -deprecation
for details
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
at
org.apache.phoenix.spark.PhoenixRDD.getPhoenixConfiguration(PhoenixRDD.scala:82)
// Or, this gets the same error
scala> val rdd = sc.phoenixTableAsRDD("EMAIL_ENRON", Seq("MAIL_FROM",
"MAIL_TO"), zkUrl=Some("localhost"))
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
at
org.apache.phoenix.spark.PhoenixRDD.getPhoenixConfiguration(PhoenixRDD.scala:82)
at
org.apache.phoenix.spark.PhoenixRDD.phoenixConf$lzycompute(PhoenixRDD.scala:38)
6. I've tried every permutation I can think of, and also spent hours
Googling. Some times I can get different errors, but always errors.
Interestingly, if I manage to load the HBaseConfiguration class
manually (by specifying classpaths and then import), I get a
"phoenixTableAsRDD is not a member of SparkContext" error.
How can I use Phoenix from within Spark? I'm really eager to do so,
but haven't been able to.
Also: Can someone give me some background on the underlying issues
here? Trial-and-error-plus-google is not exactly high quality
engineering; I'd like to understand the problem better.
