Hi,
in an already working Spark - Hive environment with Spark 1.6 and Hive 1.2.1,
with Hive metastore configured on Postgres DB, I have upgraded Spark to the
2.0.0.
I have started the thrift server on YARN, then tried to execute from the
beeline cli or a jdbc client the following command:
SHOW DATABASES;
It always gives this error on Spark server side:
spark@spark-test[spark] /home/spark> beeline -u jdbc:hive2://$(hostname):10000
-n spark
Connecting to jdbc:hive2://spark-test:10000
16/09/05 17:41:43 INFO jdbc.Utils: Supplied authorities: spark-test:10000
16/09/05 17:41:43 INFO jdbc.Utils: Resolved authority: spark-test:10000
16/09/05 17:41:43 INFO jdbc.HiveConnection: Will try to open client transport
with JDBC Uri: jdbc:hive2:// spark-test:10000
Connected to: Spark SQL (version 2.0.0)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1.spark2 by Apache Hive
0: jdbc:hive2:// spark-test:10000> show databases;
java.lang.IllegalStateException: Can't overwrite cause with
java.lang.ClassCastException:
org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to
org.apache.spark.sql.catalyst.expressions.UnsafeRow
at java.lang.Throwable.initCause(Throwable.java:457)
at
org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236)
at
org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236)
at
org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:197)
at
org.apache.hive.service.cli.HiveSQLException.<init>(HiveSQLException.java:108)
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:256)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:242)
at
org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:365)
at org.apache.hive.beeline.BufferedRows.<init>(BufferedRows.java:42)
at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
at org.apache.hive.beeline.Commands.execute(Commands.java:860)
at org.apache.hive.beeline.Commands.sql(Commands.java:713)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
at
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
Task 0 in stage 3.0 failed 10 times, most recent failure: Lost task 0.9 in
stage 3.0 (TID 12, vertica204): java.lang.ClassCastException:
org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to
org.apache.spark.sql.catalyst.expressions.UnsafeRow
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:247)
at
org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:240)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:784)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:784)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:85)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.hive.service.cli.HiveSQLException.newInstance(HiveSQLException.java:244)
at
org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:210)
... 15 more
Error: Error retrieving next row (state=,code=0)
The same command works when using Spark 1.6, is it a possible issue?
Thanks!
