[
https://issues.apache.org/jira/browse/HIVE-18655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357282#comment-16357282
]
Sahil Takiar commented on HIVE-18655:
-------------------------------------
Hive 2.1.1 was only ever tested with Spark 1.6.0 so its unlikely to work with
Spark 2.0. If you want to use Spark 2.0.0 try Hive 2.3.x which has been tested
with Spark 2.0.0.
> Apache hive 2.1.1 on Apache Spark 2.0
> -------------------------------------
>
> Key: HIVE-18655
> URL: https://issues.apache.org/jira/browse/HIVE-18655
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2, Spark
> Affects Versions: 2.1.1
> Environment: apache hive -2.1.1
> apache spark - 2.0 - prebulit version (removed hive jars)
> apache hadoop -2.8
> Reporter: AbdulMateen
> Priority: Blocker
>
>
> |Hi,
>
> when connecting my beeline in hive it is not able to create spark client.
>
> {{select count(*) from student; Query ID =
> hadoop_20180208184224_f86b5aeb-f27b-4156-bd77-0aab54c0ec67 Total jobs = 1
> Launching Job 1 out of 1 In order to change the average load for a reducer
> (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to
> limit the maximum number of reducers: set hive.exec.reducers.max=<number> In
> order to set a constant number of reducers: set
> mapreduce.job.reduces=<number> }}
> Failed to execute spark task, with exception
> 'org.apache.hadoop.hive.ql.metadata.HiveException(*Failed to create spark
> client*.)'
> { {FAILED: Execution Error, return code 1 from
> {color:#FF0000}org.apache.hadoop.hive.ql.exec.spark.SparkTask Error: Error
> while processing statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask (state=08S01,code=1) }{color}
> }
>
>
> Installed spark prebuilt 2.0 one in standalone cluster mode
>
> My hive-site.xml – placed in spark/conf too and removed the hive jars in
> hdfs path
>
> {{<property> <name>spark.master</name> <value>yarn</value> <description>
> Spark Master URL</description> </property> <property>
> <name>spark.eventLog.enabled</name> <value>true</value> <description>Spark
> Event Log</description> </property> <property>
> <name>spark.eventLog.dir</name>
> <value>hdfs://xx.xxx.xx.xx:9000/user/spark/eventLogging</value>
> <description>Spark event log folder</description> </property> <property>
> <name>spark.executor.memory</name> <value>512m</value> <description>Spark
> executor memory</description> </property> <property>
> <name>spark.serializer</name>
> <value>org.apache.spark.serializer.KryoSerializer</value> <description>Spark
> serializer</description> </property> <property> <name>spark.yarn.jars</name>
> <value>hdfs://xx.xxx.xx.xx:9000:/user/spark/spark-jars/*</value> </property>
> <property> <name>spark.submit.deployMode</name> <value>cluster</value>
> <description>Spark Master URL</description> </property }}
>
> My yarn-site.xml
>
> {{<property> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>40960</value> </property> <property>
> <name>yarn.scheduler.minimum-allocation-mb</name> <value>2048</value>
> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name>
> <value>8192</value> </property>}}|
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
