Re: setup spark engine to hive ,the hive version and spark build problem

[Mich Talebzadeh]

the only way I managed this work (and I have not tried newer build yet) was
Hive. 2.0.1 on Spark 1.3.1 engine

Connected to: Apache Hive (version 2.0.1

0: jdbc:hive2://rhes564:10099/default> select count(1) from passwd;
INFO  : Compiling
command(queryId=hduser_20170617175040_5d4a85d9-a306-4cb3-a191-e675cdd21dcf):
select count(1) from passwd
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:c0,
type:bigint, comment:null)], properties:null)
INFO  : Completed compiling
command(queryId=hduser_20170617175040_5d4a85d9-a306-4cb3-a191-e675cdd21dcf);
Time taken: 0.064 seconds
INFO  : Executing
command(queryId=hduser_20170617175040_5d4a85d9-a306-4cb3-a191-e675cdd21dcf):
select count(1) from passwd
INFO  : Query ID =
hduser_20170617175040_5d4a85d9-a306-4cb3-a191-e675cdd21dcf
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  :
*Query Hive on Spark job[0] stages:*
INFO  : 0
INFO  : 1
INFO  :
*Status: Running (Hive on Spark job[0])*

HTH


Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 17 June 2017 at 17:33, Boris Lublinsky 
wrote:

> You need to exlicitely build spark without Hive. Look at getting started
> doc
>
> Get Outlook for Android 
>
>
>
>
> On Sat, Jun 17, 2017 at 5:26 AM -0400, "wuchang" <583424...@qq.com> wrote:
>
> I want to build hive and spark to make my hive based on spark engine.I
>> choose Hive 2.3.0 and Spark 2.0.0， which is claimed to be *compatible*
>> by hive official document.According to the hive officials document ,I
>> have to build spark *without hive profile *to avoid the conflict between
>> original hive and spark-integrated hive. Yes, I build successfully , but
>> then the problem comes:*I cannot use spark-sql anymore* because *spark-sql
>> relies on the hive library and my spark is a no-hive build.*
>>
>> [appuser@ab-10-11-22-209 spark]$ spark-sql java.lang.ClassNotFoundException:
>> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver at
>> java.net.URLClassLoader.findClass(URLClassLoader.java:381) at
>> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
>> java.lang.ClassLoader.loadClass(ClassLoader.java:357) at
>> java.lang.Class.forName0(Native Method) at 
>> java.lang.Class.forName(Class.java:348)
>> at org.apache.spark.util.Utils$.classForName(Utils.scala:225) at
>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
>> deploy$SparkSubmit$$runMain(SparkSubmit.scala:686) at
>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) at
>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) at
>> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Failed to
>> load main class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.
>> You need to build Spark with *-Phive and -Phive-thriftserver.*
>> How can I build and setup spark and to make hive on spark
>> Work properly and my spark-sql、pyspark and spark-shell work properly?
>>
>>
>> I don’t know the relationship between *spark-integrated hive* and *original
>> hive. *Below is the spark-integrated hive jars:
>>
>> hive-beeline-1.2.1.spark2.jarhive-cli-1.2.1.spark2.jarhive-exec-1.2.1.spark2.jarhive-jdbc-1.2.1.spark2.jarhive-metastore-1.2.1.spark2.jarspark-hive_2.11-2.0.0.jarspark-hive-thriftserver_2.11-2.0.0.jar
>>
>>
>> It seems that Spark 2.0.0 relies on hive 1.2.1.
>>
>>
>>

Re: setup spark engine to hive ,the hive version and spark build problem

[Boris Lublinsky]

You need to exlicitely build spark without Hive. Look at getting started doc




Get Outlook for Android







On Sat, Jun 17, 2017 at 5:26 AM -0400, "wuchang" <583424...@qq.com> wrote:










I want to build hive and spark to make my hive based on spark engine.I choose 
Hive 2.3.0 and Spark 2.0.0， which is claimed to be compatible by hive official 
document.According to the hive officials document ,I  have to build spark 
without hive profile to avoid the conflict between original hive and 
spark-integrated hive. Yes, I build successfully , but then the problem comes:I 
cannot use spark-sql anymore because spark-sql relies on the hive library and 
my spark is a no-hive build.

[appuser@ab-10-11-22-209 spark]$ spark-sql
java.lang.ClassNotFoundException: 
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:686)
at 
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Failed to load main class 
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.
You need to build Spark with -Phive and -Phive-thriftserver.
How can I build and setup spark and to make hive on sparkWork properly and my 
spark-sql、pyspark and spark-shell work properly?

I don’t know the relationship between spark-integrated hive and original hive. 
Below is the spark-integrated hive jars:
hive-beeline-1.2.1.spark2.jar
hive-cli-1.2.1.spark2.jar
hive-exec-1.2.1.spark2.jar
hive-jdbc-1.2.1.spark2.jar
hive-metastore-1.2.1.spark2.jar
spark-hive_2.11-2.0.0.jar
spark-hive-thriftserver_2.11-2.0.0.jar
It seems that Spark 2.0.0 relies on hive 1.2.1.

setup spark engine to hive ,the hive version and spark build problem

[wuchang]

I want to build hive and spark to make my hive based on spark engine.
I choose Hive 2.3.0 and Spark 2.0.0， which is claimed to be compatible by hive 
official document.
According to the hive officials document ,I  have to build spark without hive 
profile to avoid the conflict between original hive and spark-integrated hive. 
Yes, I build successfully , but then the problem comes:I cannot use spark-sql 
anymore because spark-sql relies on the hive library and my spark is a no-hive 
build.


[appuser@ab-10-11-22-209 spark]$ spark-sql
java.lang.ClassNotFoundException: 
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:686)
at 
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Failed to load main class 
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.
You need to build Spark with -Phive and -Phive-thriftserver.

How can I build and setup spark and to make hive on spark
Work properly and my spark-sql、pyspark and spark-shell work properly?


I don’t know the relationship between spark-integrated hive and original hive. 
Below is the spark-integrated hive jars:

hive-beeline-1.2.1.spark2.jar
hive-cli-1.2.1.spark2.jar
hive-exec-1.2.1.spark2.jar
hive-jdbc-1.2.1.spark2.jar
hive-metastore-1.2.1.spark2.jar
spark-hive_2.11-2.0.0.jar
spark-hive-thriftserver_2.11-2.0.0.jar

It seems that Spark 2.0.0 relies on hive 1.2.1.

setup spark engine to hive ,the hive version and spark build problem

[wuchang]

I want to build hive and spark to make my hive based on spark engine.
I choose Hive 2.3.0 and Spark 2.0.0， which is claimed to be compatible by hive 
official document.
According to the hive officials document ,I  have to build spark without hive 
profile to avoid the conflict between original hive and spark-integrated hive. 
Yes, I build successfully , but then the problem comes:I cannot use spark-sql 
anymore because spark-sql relies on the hive library and my spark is a no-hive 
build.

I don’t know the relationship between hive-integrated hive and original hive. 
Below is the spark-integrated hive jars:

hive-beeline-1.2.1.spark2.jar
hive-cli-1.2.1.spark2.jar
hive-exec-1.2.1.spark2.jar
hive-jdbc-1.2.1.spark2.jar
hive-metastore-1.2.1.spark2.jar
spark-hive_2.11-2.0.0.jar
spark-hive-thriftserver_2.11-2.0.0.jar

It seems that Spark 2.0.0 relies on hive 1.2.1.
How can I build and setup spark and to make hive on spark
Work properly and my spark-sql、pyspark and spark-shell work properly?