RE: Spark 2.0.0 Thrift Server problem with Hive metastore

2016-09-06 Thread Campagnola, Francesco 
The same error occurs when executing any “explain” command:

0: jdbc:hive2://spark-test:1> explain select 1 as id;
java.lang.IllegalStateException: Can't overwrite cause with 
java.lang.ClassCastException: 
org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to 
org.apache.spark.sql.catalyst.expressions.UnsafeRow
…
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: 
Task 0 in stage 198.0 failed 10 times, most recent failure: Lost task 0.9 in 
stage 198.0 (TID 2046, vertica204): java.lang.ClassCastException: 
org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to 
org.apache.spark.sql.catalyst.expressions.UnsafeRow
at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:247)

I have checked the source code, and it seems this explicit cast is causing the 
issue:

>spark-2.0.0/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala

private def getByteArrayRdd(n: Int = -1): RDD[Array[Byte]] = {
execute().mapPartitionsInternal { iter =>
  var count = 0
  val buffer = new Array[Byte](4 << 10)  // 4K
  val codec = CompressionCodec.createCodec(SparkEnv.get.conf)
  val bos = new ByteArrayOutputStream()
  val out = new DataOutputStream(codec.compressedOutputStream(bos))
  while (iter.hasNext && (n < 0 || count < n)) {
val row = iter.next().asInstanceOf[UnsafeRow]

Hope this issue will be fixed in the next releases…


From: Campagnola, Francesco
Sent: martedì 6 settembre 2016 09:46
To: 'Jeff Zhang' <zjf...@gmail.com>
Cc: user@spark.apache.org
Subject: RE: Spark 2.0.0 Thrift Server problem with Hive metastore

I mean I have installed Spark 2.0 in the same environment where Spark 1.6 
thrift server was running,
then stopped the Spark 1.6 thrift server and started the Spark 2.0 one.

If I’m not mistaken, Spark 2.0 should be still compatible with Hive 1.2.1 and 
no upgrade procedures are required.
The spark-defaults.conf file has not been changed.

The following commands issued to the Spark 2.0 thrift server work:
create database test;
use test;
create table tb_1 (id int);
insert into table tb_1 select t.id from (select 1 as id) t;

While all of these commands return the same error:
show databases;
show tables;
show partitions tb_1;
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: 
Task 0 in stage 62.0 failed 10 times, most recent failure: Lost task 0.9 in 
stage 62.0 (TID 540, vertica204): java.lang.ClassCastException: 
org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to 
org.apache.spark.sql.catalyst.expressions.UnsafeRow





From: Jeff Zhang [mailto:zjf...@gmail.com]
Sent: martedì 6 settembre 2016 02:50
To: Campagnola, Francesco 
<francesco.campagn...@anritsu.com<mailto:francesco.campagn...@anritsu.com>>
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: Spark 2.0.0 Thrift Server problem with Hive metastore

How do you upgrade to spark 2.0 ?

On Mon, Sep 5, 2016 at 11:25 PM, Campagnola, Francesco 
<francesco.campagn...@anritsu.com<mailto:francesco.campagn...@anritsu.com>> 
wrote:
Hi,

in an already working Spark - Hive environment with Spark 1.6 and Hive 1.2.1, 
with Hive metastore configured on Postgres DB, I have upgraded Spark to the 
2.0.0.

I have started the thrift server on YARN, then tried to execute from the 
beeline cli or a jdbc client the following command:
SHOW DATABASES;
It always gives this error on Spark server side:

spark@spark-test[spark] /home/spark> beeline -u jdbc:hive2://$(hostname):1 
-n spark

Connecting to jdbc:hive2://spark-test:1
16/09/05 17:41:43 INFO jdbc.Utils: Supplied authorities: spark-test:1
16/09/05 17:41:43 INFO jdbc.Utils: Resolved authority: spark-test:1
16/09/05 17:41:43 INFO jdbc.HiveConnection: Will try to open client transport 
with JDBC Uri: jdbc:hive2:// spark-test:1
Connected to: Spark SQL (version 2.0.0)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1.spark2 by Apache Hive

0: jdbc:hive2:// spark-test:1> show databases;
java.lang.IllegalStateException: Can't overwrite cause with 
java.lang.ClassCastException: 
org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to 
org.apache.spark.sql.catalyst.expressions.UnsafeRow
at java.lang.Throwable.initCause(Throwable.java:457)
at 
org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236)
at 
org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236)
at 
org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:197)
at 
org.apache.hive.service.cli.HiveSQLException.(HiveSQLException.java:108)
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:256)

Re: Spark 2.0.0 Thrift Server problem with Hive metastore

2016-09-06 Thread Chanh Le 
Did anyone use STS of Spark 2.0 on production?
For me I still waiting for the compatible in parquet file created by Spark 
1.6.1 


> On Sep 6, 2016, at 2:46 PM, Campagnola, Francesco 
> <francesco.campagn...@anritsu.com> wrote:
> 
> I mean I have installed Spark 2.0 in the same environment where Spark 1.6 
> thrift server was running,
> then stopped the Spark 1.6 thrift server and started the Spark 2.0 one.
>  
> If I’m not mistaken, Spark 2.0 should be still compatible with Hive 1.2.1 and 
> no upgrade procedures are required.
> The spark-defaults.conf file has not been changed.
>  
> The following commands issued to the Spark 2.0 thrift server work:
> create database test;
> use test;
> create table tb_1 (id int);
> insert into table tb_1 select t.id from (select 1 as id) t;
>  
> While all of these commands return the same error:
> show databases;
> show tables;
> show partitions tb_1;
> Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: 
> Task 0 in stage 62.0 failed 10 times, most recent failure: Lost task 0.9 in 
> stage 62.0 (TID 540, vertica204): java.lang.ClassCastException: 
> org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast 
> to org.apache.spark.sql.catalyst.expressions.UnsafeRow
>  
>  
>  
>  
> From: Jeff Zhang [mailto:zjf...@gmail.com <mailto:zjf...@gmail.com>] 
> Sent: martedì 6 settembre 2016 02:50
> To: Campagnola, Francesco <francesco.campagn...@anritsu.com 
> <mailto:francesco.campagn...@anritsu.com>>
> Cc: user@spark.apache.org <mailto:user@spark.apache.org>
> Subject: Re: Spark 2.0.0 Thrift Server problem with Hive metastore
>  
> How do you upgrade to spark 2.0 ? 
>  
> On Mon, Sep 5, 2016 at 11:25 PM, Campagnola, Francesco 
> <francesco.campagn...@anritsu.com <mailto:francesco.campagn...@anritsu.com>> 
> wrote:
> Hi,
>  
> in an already working Spark - Hive environment with Spark 1.6 and Hive 1.2.1, 
> with Hive metastore configured on Postgres DB, I have upgraded Spark to the 
> 2.0.0.
>  
> I have started the thrift server on YARN, then tried to execute from the 
> beeline cli or a jdbc client the following command:
> SHOW DATABASES;
> It always gives this error on Spark server side:
>  
> spark@spark-test[spark] /home/spark> beeline -u 
> jdbc:hive2://$(hostname):1 -n spark
>  
> Connecting to jdbc:hive2://spark-test:1
> 16/09/05 17:41:43 INFO jdbc.Utils: Supplied authorities: spark-test:1
> 16/09/05 17:41:43 INFO jdbc.Utils: Resolved authority: spark-test:1
> 16/09/05 17:41:43 INFO jdbc.HiveConnection: Will try to open client transport 
> with JDBC Uri: jdbc:hive2:// spark-test:1
> Connected to: Spark SQL (version 2.0.0)
> Driver: Hive JDBC (version 1.2.1.spark2)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 1.2.1.spark2 by Apache Hive
>  
> 0: jdbc:hive2:// spark-test:1> show databases;
> java.lang.IllegalStateException: Can't overwrite cause with 
> java.lang.ClassCastException: 
> org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast 
> to org.apache.spark.sql.catalyst.expressions.UnsafeRow
> at java.lang.Throwable.initCause(Throwable.java:457)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236)
> at 
> org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:197)
> at 
> org.apache.hive.service.cli.HiveSQLException.(HiveSQLException.java:108)
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:256)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:242)
> at 
> org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:365)
> at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:42)
> at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
> at org.apache.hive.beeline.Commands.execute(Commands.java:860)
> at org.apache.hive.beeline.Commands.sql(Commands.java:713)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
> at 
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
> Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: 
> Task 0 in stage 3.0 failed 10 times, most recent failure: Lost task 0.9 in 
> stage 3.0 (TID 12, vertica204): java.lang.ClassC

RE: Spark 2.0.0 Thrift Server problem with Hive metastore

2016-09-06 Thread Campagnola, Francesco 
I mean I have installed Spark 2.0 in the same environment where Spark 1.6 
thrift server was running,
then stopped the Spark 1.6 thrift server and started the Spark 2.0 one.

If I’m not mistaken, Spark 2.0 should be still compatible with Hive 1.2.1 and 
no upgrade procedures are required.
The spark-defaults.conf file has not been changed.

The following commands issued to the Spark 2.0 thrift server work:
create database test;
use test;
create table tb_1 (id int);
insert into table tb_1 select t.id from (select 1 as id) t;

While all of these commands return the same error:
show databases;
show tables;
show partitions tb_1;
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: 
Task 0 in stage 62.0 failed 10 times, most recent failure: Lost task 0.9 in 
stage 62.0 (TID 540, vertica204): java.lang.ClassCastException: 
org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to 
org.apache.spark.sql.catalyst.expressions.UnsafeRow




From: Jeff Zhang [mailto:zjf...@gmail.com]
Sent: martedì 6 settembre 2016 02:50
To: Campagnola, Francesco <francesco.campagn...@anritsu.com>
Cc: user@spark.apache.org
Subject: Re: Spark 2.0.0 Thrift Server problem with Hive metastore

How do you upgrade to spark 2.0 ?

On Mon, Sep 5, 2016 at 11:25 PM, Campagnola, Francesco 
<francesco.campagn...@anritsu.com<mailto:francesco.campagn...@anritsu.com>> 
wrote:
Hi,

in an already working Spark - Hive environment with Spark 1.6 and Hive 1.2.1, 
with Hive metastore configured on Postgres DB, I have upgraded Spark to the 
2.0.0.

I have started the thrift server on YARN, then tried to execute from the 
beeline cli or a jdbc client the following command:
SHOW DATABASES;
It always gives this error on Spark server side:

spark@spark-test[spark] /home/spark> beeline -u jdbc:hive2://$(hostname):1 
-n spark

Connecting to jdbc:hive2://spark-test:1
16/09/05 17:41:43 INFO jdbc.Utils: Supplied authorities: spark-test:1
16/09/05 17:41:43 INFO jdbc.Utils: Resolved authority: spark-test:1
16/09/05 17:41:43 INFO jdbc.HiveConnection: Will try to open client transport 
with JDBC Uri: jdbc:hive2:// spark-test:1
Connected to: Spark SQL (version 2.0.0)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1.spark2 by Apache Hive

0: jdbc:hive2:// spark-test:1> show databases;
java.lang.IllegalStateException: Can't overwrite cause with 
java.lang.ClassCastException: 
org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to 
org.apache.spark.sql.catalyst.expressions.UnsafeRow
at java.lang.Throwable.initCause(Throwable.java:457)
at 
org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236)
at 
org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236)
at 
org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:197)
at 
org.apache.hive.service.cli.HiveSQLException.(HiveSQLException.java:108)
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:256)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:242)
at 
org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:365)
at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:42)
at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
at org.apache.hive.beeline.Commands.execute(Commands.java:860)
at org.apache.hive.beeline.Commands.sql(Commands.java:713)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
at 
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: 
Task 0 in stage 3.0 failed 10 times, most recent failure: Lost task 0.9 in 
stage 3.0 (TID 12, vertica204): java.lang.ClassCastException: 
org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to 
org.apache.spark.sql.catalyst.expressions.UnsafeRow
at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:247)
at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:240)
at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:784)
at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:784)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at org.apache.spark.scheduler.ResultTask.run

Re: Spark 2.0.0 Thrift Server problem with Hive metastore

2016-09-05 Thread Jeff Zhang 
How do you upgrade to spark 2.0 ?

On Mon, Sep 5, 2016 at 11:25 PM, Campagnola, Francesco <
francesco.campagn...@anritsu.com> wrote:

> Hi,
>
>
>
> in an already working Spark - Hive environment with Spark 1.6 and Hive
> 1.2.1, with Hive metastore configured on Postgres DB, I have upgraded Spark
> to the 2.0.0.
>
>
>
> I have started the thrift server on YARN, then tried to execute from the
> beeline cli or a jdbc client the following command:
>
> SHOW DATABASES;
>
> It always gives this error on Spark server side:
>
>
>
> spark@spark-test[spark] /home/spark> beeline -u
> jdbc:hive2://$(hostname):1 -n spark
>
>
>
> Connecting to jdbc:hive2://spark-test:1
>
> 16/09/05 17:41:43 INFO jdbc.Utils: Supplied authorities: spark-test:1
>
> 16/09/05 17:41:43 INFO jdbc.Utils: Resolved authority: spark-test:1
>
> 16/09/05 17:41:43 INFO jdbc.HiveConnection: Will try to open client
> transport with JDBC Uri: jdbc:hive2:// spark-test:1
>
> Connected to: Spark SQL (version 2.0.0)
>
> Driver: Hive JDBC (version 1.2.1.spark2)
>
> Transaction isolation: TRANSACTION_REPEATABLE_READ
>
> Beeline version 1.2.1.spark2 by Apache Hive
>
>
>
> 0: jdbc:hive2:// spark-test:1> show databases;
>
> java.lang.IllegalStateException: Can't overwrite cause with
> java.lang.ClassCastException: 
> org.apache.spark.sql.catalyst.expressions.GenericInternalRow
> cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow
>
> at java.lang.Throwable.initCause(Throwable.java:457)
>
> at org.apache.hive.service.cli.HiveSQLException.toStackTrace(
> HiveSQLException.java:236)
>
> at org.apache.hive.service.cli.HiveSQLException.toStackTrace(
> HiveSQLException.java:236)
>
> at org.apache.hive.service.cli.HiveSQLException.toCause(
> HiveSQLException.java:197)
>
> at org.apache.hive.service.cli.HiveSQLException.(
> HiveSQLException.java:108)
>
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:256)
>
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.
> java:242)
>
> at org.apache.hive.jdbc.HiveQueryResultSet.next(
> HiveQueryResultSet.java:365)
>
> at org.apache.hive.beeline.BufferedRows.(
> BufferedRows.java:42)
>
> at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794)
>
> at org.apache.hive.beeline.Commands.execute(Commands.java:860)
>
> at org.apache.hive.beeline.Commands.sql(Commands.java:713)
>
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973)
>
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813)
>
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771)
>
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(
> BeeLine.java:484)
>
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
>
> Caused by: org.apache.spark.SparkException: Job aborted due to stage
> failure: Task 0 in stage 3.0 failed 10 times, most recent failure: Lost
> task 0.9 in stage 3.0 (TID 12, vertica204): java.lang.ClassCastException:
> org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be
> cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow
>
> at org.apache.spark.sql.execution.SparkPlan$$anonfun$
> 4.apply(SparkPlan.scala:247)
>
> at org.apache.spark.sql.execution.SparkPlan$$anonfun$
> 4.apply(SparkPlan.scala:240)
>
> at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$
> 1$$anonfun$apply$24.apply(RDD.scala:784)
>
> at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$
> 1$$anonfun$apply$24.apply(RDD.scala:784)
>
> at org.apache.spark.rdd.MapPartitionsRDD.compute(
> MapPartitionsRDD.scala:38)
>
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
>
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.
> scala:70)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:85)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(
> Executor.scala:274)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
>
> Driver stacktrace:
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:62)
>
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>
> at org.apache.hive.service.cli.HiveSQLException.newInstance(
> HiveSQLException.java:244)
>
> at org.apache.hive.service.cli.HiveSQLException.toStackTrace(
> HiveSQLException.java:210)
>
> ... 15 more
>
> Error: Error