[
https://issues.apache.org/jira/browse/HIVE-9970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tarush Grover reopened HIVE-9970:
---------------------------------
Assignee: Tarush Grover
This issue still persists my environment is :
hadoop = 2.6
hive = 1.1.1
spark = 1.5.1
below are logs in the hive.log :
15/11/21 19:56:27 [HiveServer2-Background-Pool: Thread-39]: ERROR exec.Task:
Failed to execute spark task, with exception
'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark
client.)'
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client.
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:57)
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
at
org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:113)
at
org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:95)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1638)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1397)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:145)
at
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:70)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:209)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot run program
"/home/adt/server/spark1.5/spark-1.5.1set
mapreduce.input.fileinputformat.split.maxsize=750000000set
hive.vectorized.execution.enabled=trueset hive.cbo.enable=trueset
hive.optimize.reducededuplication.min.reducer=4set
hive.optimize.reducededuplication=trueset
hive.orc.splits.include.file.footer=falseset hive.merge.mapfiles=trueset
hive.merge.sparkfiles=falseset hive.merge.smallfiles.avgsize=16000000set
hive.merge.size.per.task=256000000set hive.merge.orcfile.stripe.level=trueset
hive.auto.convert.join=trueset hive.auto.convert.join.noconditionaltask=trueset
hive.auto.convert.join.noconditionaltask.size=894435328set
hive.optimize.bucketmapjoin.sortedmerge=falseset
hive.map.aggr.hash.percentmemory=0.5set hive.map.aggr=trueset
hive.optimize.sort.dynamic.partition=falseset hive.stats.autogather=trueset
hive.stats.fetch.column.stats=trueset
hive.vectorized.execution.reduce.enabled=falseset
hive.vectorized.groupby.checkinterval=4096set
hive.vectorized.groupby.flush.percent=0.1set
hive.compute.query.using.stats=trueset hive.limit.pushdown.memory.usage=0.4set
hive.optimize.index.filter=trueset
hive.exec.reducers.bytes.per.reducer=67108864set
hive.smbjoin.cache.rows=10000set hive.exec.orc.default.stripe.size=67108864set
hive.fetch.task.conversion=moreset
hive.fetch.task.conversion.threshold=1073741824set
hive.fetch.task.aggr=falseset
mapreduce.input.fileinputformat.list-status.num-threads=5set
spark.kryo.referenceTracking=false#set
spark.kryo.classesToRegister=org.apache.hadoop.hive.ql.io.HiveKey,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch/bin/spark-submit":
error=36, File name too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at
org.apache.hive.spark.client.SparkClientImpl.startDriver(SparkClientImpl.java:376)
at
org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:89)
at
org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80)
at
org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:88)
at
org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:58)
at
org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
... 22 more
Caused by: java.io.IOException: error=36, File name too long
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 28 more
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask
> Hive on spark
> -------------
>
> Key: HIVE-9970
> URL: https://issues.apache.org/jira/browse/HIVE-9970
> Project: Hive
> Issue Type: Bug
> Reporter: Amithsha
> Assignee: Tarush Grover
>
> Hi all,
> Recently i have configured Spark 1.2.0 and my environment is hadoop
> 2.6.0 hive 1.1.0 Here i have tried hive on Spark while executing
> insert into i am getting the following g error.
> Query ID = hadoop2_20150313162828_8764adad-a8e4-49da-9ef5-35e4ebd6bc63
> Total jobs = 1
> Launching Job 1 out of 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=<number>
> Failed to execute spark task, with exception
> 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create
> spark client.)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> Have added the spark-assembly jar in hive lib
> And also in hive console using the command add jar followed by the steps
> set spark.home=/opt/spark-1.2.1/;
> add jar
> /opt/spark-1.2.1/assembly/target/scala-2.10/spark-assembly-1.2.1-hadoop2.4.0.jar;
> set hive.execution.engine=spark;
> set spark.master=spark://xxxxxxx:7077;
> set spark.eventLog.enabled=true;
> set spark.executor.memory=512m;
> set spark.serializer=org.apache.spark.serializer.KryoSerializer;
> Can anyone suggest!!!!
> Thanks & Regards
> Amithsha
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
