hello，
I tried to use branch-1.1 under hdp2.6.0-spark2.1.0 and met so many
problems.
1.build: use "mvn package -DskipTests -Pspark-2.1 -Dspark.version=2.1.0
-Phadoop-2.7.2" and
while compiling core module,throw class not found error,like
org.apache.thrift.TBase,
modify pom.xml in core and add libthrift dependency,compile
successfully.
2.running in spark-shell like ;
spark-shell --jars
carbondata_2.11-1.1.1-SNAPSHOT-shade-hadoop2.7.2.jar
scala>import org.apache.spark.sql.SparkSession
scala>import org.apache.spark.sql.CarbonSession._
scala>val cc =
SparkSession.builder().config(sc.getConf).getOrCreateCarbonSession(hdfs path)
and got error:
java.lang.NoClassDefFoundError:
org/apache/spark/sql/catalyst/CatalystConf
at
org.apache.spark.sql.hive.CarbonSessionState.analyzer$lzycompute(CarbonSessionState.scala:127)
at
org.apache.spark.sql.hive.CarbonSessionState.analyzer(CarbonSessionState.scala:126)
at
org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:69)
at
org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:67)
at
org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:50)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
... 52 elided
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.sql.catalyst.CatalystConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 59 more
check spark-catalyst.jar and didn't found class CatalystConf
3.use apache-spark catalyst jars and rerun with
spark-shell --jars
carbondata_2.11-1.1.1-SNAPSHOT-shade-hadoop2.7.2.jar,spark-catalyst_2.11-2.1.0.jar
and got error message like:
java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/CatalystConf
at
org.apache.spark.sql.hive.CarbonSessionState.analyzer$lzycompute(CarbonSessionState.scala:127)
at
org.apache.spark.sql.hive.CarbonSessionState.analyzer(CarbonSessionState.scala:126)
at
org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:69)
at
org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:67)
at
org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:50)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:63)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
... 52 elided
Caused by: java.lang.ClassNotFoundException:
org.apache.spark.sql.catalyst.CatalystConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 59 more
[email protected]
