Re: Spark1.3.1 build issue with CDH5.4.0 getUnknownFields
I had already tested query in Hive CLI and it works fine. Same query shows error in Spark SQL. On May 29, 2015 4:14 AM, ayan guha guha.a...@gmail.com wrote: Probably a naive question: can you try the same in hive CLI and see if your SQL is working? Looks like hive thing to me as spark is faithfully delegating the query to hive. On 29 May 2015 03:22, Abhishek Tripathi trackissue...@gmail.com wrote: Hi , I'm using CDH5.4.0 quick start VM and tried to build Spark with Hive compatibility so that I can run Spark sql and access temp table remotely. I used below command to build Spark, it was build successful but when I tried to access Hive data from Spark sql, I get error. Thanks, Abhi --- mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.4.0 -Phive -Phive-thriftserver -DskipTests clean package [cloudera@quickstart spark-1.3.1]$export HADOOP_CONF_DIR=/etc/hive/conf/ [cloudera@quickstart spark-1.3.1]$ ./bin/spark-sql SET spark.sql.hive.version=0.13.1 spark-sql show tables; sample_07 false t1 false Time taken: 3.901 seconds, Fetched 2 row(s) spark-sql select * from t1; 15/05/19 23:48:46 ERROR SparkSQLDriver: Failed in [select * from t1] java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$SetOwnerRequestProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
Re: Spark1.3.1 build issue with CDH5.4.0 getUnknownFields
Regarding the build itself, hadoop-2.6 is not even a valid profile. I got the following WARNING for my build. [WARNING] The requested profile hadoop-2.6 could not be activated because it does not exist. Chen On Fri, May 29, 2015 at 2:38 AM, trackissue121 trackissue...@gmail.com wrote: I had already tested query in Hive CLI and it works fine. Same query shows error in Spark SQL. On May 29, 2015 4:14 AM, ayan guha guha.a...@gmail.com wrote: Probably a naive question: can you try the same in hive CLI and see if your SQL is working? Looks like hive thing to me as spark is faithfully delegating the query to hive. On 29 May 2015 03:22, Abhishek Tripathi trackissue...@gmail.com wrote: Hi , I'm using CDH5.4.0 quick start VM and tried to build Spark with Hive compatibility so that I can run Spark sql and access temp table remotely. I used below command to build Spark, it was build successful but when I tried to access Hive data from Spark sql, I get error. Thanks, Abhi --- *mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.4.0 -Phive -Phive-thriftserver -DskipTests clean package* [cloudera@quickstart spark-1.3.1]$export HADOOP_CONF_DIR=/etc/hive/conf/ [cloudera@quickstart spark-1.3.1]$ ./bin/spark-sql SET spark.sql.hive.version=0.13.1 spark-sql show tables; sample_07 false t1 false Time taken: 3.901 seconds, Fetched 2 row(s) spark-sql select * from t1; 15/05/19 23:48:46 ERROR SparkSQLDriver: Failed in [select * from t1] java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$SetOwnerRequestProto overrides final method *getUnknownFields* .()Lcom/google/protobuf/UnknownFieldSet; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) -- Chen Song
Re: Spark1.3.1 build issue with CDH5.4.0 getUnknownFields
I've gotten that error when something is trying to use a different version of protobuf than you want. Maybe check out a `mvn dependency:tree` to see if someone is trying to use something other than libproto 2.5.0. (At least, 2.5.0 was current when I was having the problem) On Fri, May 29, 2015 at 10:23 AM, Chen Song chen.song...@gmail.com wrote: Regarding the build itself, hadoop-2.6 is not even a valid profile. I got the following WARNING for my build. [WARNING] The requested profile hadoop-2.6 could not be activated because it does not exist. Chen On Fri, May 29, 2015 at 2:38 AM, trackissue121 trackissue...@gmail.com wrote: I had already tested query in Hive CLI and it works fine. Same query shows error in Spark SQL. On May 29, 2015 4:14 AM, ayan guha guha.a...@gmail.com wrote: Probably a naive question: can you try the same in hive CLI and see if your SQL is working? Looks like hive thing to me as spark is faithfully delegating the query to hive. On 29 May 2015 03:22, Abhishek Tripathi trackissue...@gmail.com wrote: Hi , I'm using CDH5.4.0 quick start VM and tried to build Spark with Hive compatibility so that I can run Spark sql and access temp table remotely. I used below command to build Spark, it was build successful but when I tried to access Hive data from Spark sql, I get error. Thanks, Abhi --- *mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.4.0 -Phive -Phive-thriftserver -DskipTests clean package* [cloudera@quickstart spark-1.3.1]$export HADOOP_CONF_DIR=/etc/hive/conf/ [cloudera@quickstart spark-1.3.1]$ ./bin/spark-sql SET spark.sql.hive.version=0.13.1 spark-sql show tables; sample_07 false t1 false Time taken: 3.901 seconds, Fetched 2 row(s) spark-sql select * from t1; 15/05/19 23:48:46 ERROR SparkSQLDriver: Failed in [select * from t1] java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$SetOwnerRequestProto overrides final method *getUnknownFields* .()Lcom/google/protobuf/UnknownFieldSet; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) -- Chen Song
Re: Spark1.3.1 build issue with CDH5.4.0 getUnknownFields
Probably a naive question: can you try the same in hive CLI and see if your SQL is working? Looks like hive thing to me as spark is faithfully delegating the query to hive. On 29 May 2015 03:22, Abhishek Tripathi trackissue...@gmail.com wrote: Hi , I'm using CDH5.4.0 quick start VM and tried to build Spark with Hive compatibility so that I can run Spark sql and access temp table remotely. I used below command to build Spark, it was build successful but when I tried to access Hive data from Spark sql, I get error. Thanks, Abhi --- *mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.4.0 -Phive -Phive-thriftserver -DskipTests clean package* [cloudera@quickstart spark-1.3.1]$export HADOOP_CONF_DIR=/etc/hive/conf/ [cloudera@quickstart spark-1.3.1]$ ./bin/spark-sql SET spark.sql.hive.version=0.13.1 spark-sql show tables; sample_07 false t1 false Time taken: 3.901 seconds, Fetched 2 row(s) spark-sql select * from t1; 15/05/19 23:48:46 ERROR SparkSQLDriver: Failed in [select * from t1] java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$SetOwnerRequestProto overrides final method *getUnknownFields* .()Lcom/google/protobuf/UnknownFieldSet; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
Spark1.3.1 build issue with CDH5.4.0 getUnknownFields
Hi , I'm using CDH5.4.0 quick start VM and tried to build Spark with Hive compatibility so that I can run Spark sql and access temp table remotely. I used below command to build Spark, it was build successful but when I tried to access Hive data from Spark sql, I get error. Thanks, Abhi --- *mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.4.0 -Phive -Phive-thriftserver -DskipTests clean package* [cloudera@quickstart spark-1.3.1]$export HADOOP_CONF_DIR=/etc/hive/conf/ [cloudera@quickstart spark-1.3.1]$ ./bin/spark-sql SET spark.sql.hive.version=0.13.1 spark-sql show tables; sample_07 false t1 false Time taken: 3.901 seconds, Fetched 2 row(s) spark-sql select * from t1; 15/05/19 23:48:46 ERROR SparkSQLDriver: Failed in [select * from t1] java.lang.VerifyError: class org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$SetOwnerRequestProto overrides final method *getUnknownFields* .()Lcom/google/protobuf/UnknownFieldSet; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
