Hello, Zeppelin mailing list members,
I have installed Zeppelin on the CDH 5.4.3 Hadoop cluster, but couldn't run
spark due to the NoClassDefFoundError for HiveConf.
...
15/10/26 19:38:29 WARN shortcircuit.DomainSocketFactory: The short-circuit
local reads feature cannot be used because libhadoop cannot be loaded.
15/10/26 19:38:30 INFO scheduler.EventLoggingListener: Logging events to
hdfs://mycluster/user/spark/applicationHistory/local-1445855908613
15/10/26 19:38:30 ERROR scheduler.Job: Job failed
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2663)
at java.lang.Class.getConstructor0(Class.java:3067)
at java.lang.Class.getConstructor(Class.java:1817)
at
org.apache.zeppelin.spark.SparkInterpreter.getSQLContext(SparkInterpreter.java:210)
at
org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:476)
...
Hive jar files are already added in conf/zeppelin-env.sh by setting the
following
ZEPPELIN_CLASSPATH property.
export ZEPPELIN_CLASSPATH="/usr/lib/hive/lib/*"
After digging the problem a few hours, I found that bin/interpreter.sh
clears ZEPPELIN_CLASSPATH
before attaching SPARK_APP_JAR to it.
# set spark related env variables
if [[ "${INTERPRETER_ID}" == "spark" ]]; then
if [[ -n "${SPARK_HOME}" ]]; then
export SPARK_SUBMIT="${SPARK_HOME}/bin/spark-submit"
SPARK_APP_JAR="$(ls
${ZEPPELIN_HOME}/interpreter/spark/zeppelin-spark*.jar)"
# This will evantually passes SPARK_APP_JAR to classpath of SparkIMain
ZEPPELIN_CLASSPATH=${SPARK_APP_JAR}
...
And changing it to ZEPPELIN_CLASSPATH+=":${SPARK_APP_JAR}" solve the
problem.
I am wonder whether this is a bug or something necessary.
Best wishes,
Han-Cheol
--
Han-Cheol Cho (Ph.D)
Homepage: https://sites.google.com/site/priancho/
