Thank you, Sofia.
From the log, looks it is java.lang.AbstractMethodError that leads to the job
failure.
I am using Hive 1.2.1+Spark 1.5.2, is this a compatibility issue?
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkBuildPlan
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkCreateTran.Map 1
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO Configuration.deprecation: mapred.task.is.map is deprecated.
Instead, use mapreduce.task.ismap
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO exec.Utilities: Processing alias t1
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO exec.Utilities: Adding input file
hdfs://hadoop.bit.com:9000/user/hive/warehouse/db_1.db/t1
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: <PERFLOG method=serializePlan
from=org.apache.hadoop.hive.ql.exec.Utilities>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO exec.Utilities: Serializing MapWork via kryo
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: </PERFLOG method=serializePlan
start=1452505326307 end=1452505326332 duration=25
from=org.apache.hadoop.hive.ql.exec.Utilities>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO storage.MemoryStore: ensureFreeSpace(505800) called with
curMem=0, maxMem=278302556
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory
(estimated size 493.9 KB, free 264.9 MB)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO storage.MemoryStore: ensureFreeSpace(42143) called with
curMem=505800, maxMem=278302556
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in
memory (estimated size 41.2 KB, free 264.9 MB)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on
10.12.167.42:38883 (size: 41.2 KB, free: 265.4 MB)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 ERROR util.Utils: uncaught error in thread SparkListenerBus, stopping
SparkContext
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl:
java.lang.AbstractMethodError
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:62)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO spark.SparkContext: Created broadcast 0 from hadoopRDD at
SparkPlanGenerator.java:188
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/metrics/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/api,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/static,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/threadDump,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/environment/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/environment,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/rdd,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/pool/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/pool,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/job/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/job,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: </PERFLOG method=SparkCreateTran.Map 1
start=1452505326274 end=1452505326824 duration=550
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkCreateTran.Reducer 2
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: <PERFLOG method=serializePlan
from=org.apache.hadoop.hive.ql.exec.Utilities>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO exec.Utilities: Serializing ReduceWork via kryo
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO ui.SparkUI: Stopped Spark web UI at http://10.12.167.42:4040
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO scheduler.DAGScheduler: Stopping DAGScheduler
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO cluster.YarnClientSchedulerBackend: Asking each executor to shut
down
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO cluster.YarnClientSchedulerBackend: Stopped
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: </PERFLOG method=serializePlan
start=1452505326849 end=1452505326874 duration=25
from=org.apache.hadoop.hive.ql.exec.Utilities>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: </PERFLOG method=SparkCreateTran.Reducer 2
start=1452505326824 end=1452505326936 duration=112
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: </PERFLOG method=SparkBuildPlan
start=1452505326273 end=1452505326936 duration=663
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkBuildRDDGraph
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlan>
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:07 INFO log.PerfLogger: </PERFLOG method=SparkBuildRDDGraph
start=1452505326936 end=1452505327000 duration=64
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlan>
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:07 INFO client.RemoteDriver: Failed to run job
b5a8886e-8229-4773-b943-b3c6f3c710bf
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl:
java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.SparkContext.org$apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.SparkContext.submitJob(SparkContext.scala:1981)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1.apply(AsyncRDDActions.scala:118)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1.apply(AsyncRDDActions.scala:116)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.AsyncRDDActions.foreachAsync(AsyncRDDActions.scala:116)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.api.java.JavaRDDLike$class.foreachAsync(JavaRDDLike.scala:690)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.api.java.AbstractJavaRDDLike.foreachAsync(JavaRDDLike.scala:47)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:257)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:366)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:335)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
java.util.concurrent.FutureTask.run(FutureTask.java:262)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
java.lang.Thread.run(Thread.java:745)
16/01/11 17:42:07 [RPC-Handler-3]: INFO client.SparkClientImpl: Received result
for b5a8886e-8229-4773-b943-b3c6f3c710bf
Status: Failed
16/01/11 17:42:07 [main]: ERROR status.SparkJobMonitor: Status: Failed
At 2016-01-11 17:17:51, "Sofia" <sofia.panagiot...@taiger.com> wrote:
Check log files to see the exact error. Alternatively run hive like this, to
get the exact error
hive --hiveconf hive.root.logger=INFO,console
There is a good chance that you are encountering problems between the Hive and
Spark versions and installation.
See
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
On 11 Jan 2016, at 08:47, Todd <bit1...@163.com> wrote:
Hive,
I am trying out the Hive on Spark with hive 1.2.1 and spark 1.5.2. Could
someone help me on this? Thanks!
Following are my steps:
1. build spark 1.5.2 without Hive and Hive Thrift Server. At this point, I can
use it to submit application using spark-submit --master yarn-client
2. And the built spark assembly jar into $HIVE_HOME/lib
3. start hive and add the following parameters
hive > set spark.master=yarn-client
hive > set spark.executor.memory=512M
hive > set spark.driver.memory=512M
hive > set spark.executor.instances=1
4. Then I run a simple query : select count(1) from t1;
The job fails will following error:
===============================================================================
YARN executor launch context:
env:
CLASSPATH ->
{{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
SPARK_LOG_URL_STDERR ->
http://hadoop-Aspire-TC-606:8042/node/containerlogs/container_1452320323183_0007_01_000003/hadoop/stderr?start=-4096
SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1452320323183_0007
SPARK_YARN_CACHE_FILES_FILE_SIZES -> 142746538
SPARK_USER -> hadoop
SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE
SPARK_YARN_MODE -> true
SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1452496343550
SPARK_LOG_URL_STDOUT ->
http://hadoop-Aspire-TC-606:8042/node/containerlogs/container_1452320323183_0007_01_000003/hadoop/stdout?start=-4096
SPARK_YARN_CACHE_FILES ->
hdfs://hadoop.bit.com:9000/user/hadoop/.sparkStaging/application_1452320323183_0007/spark-assembly-1.5.2-hadoop2.6.0.jar#__spark__.jar
command:
{{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms512m
-Xmx512m
'-Dhive.spark.log.dir=/home/hadoop/software/bigdata/spark-1.5.2-bin-hadoop2.6.0-withouthive/logs/'
-Djava.io.tmpdir={{PWD}}/tmp '-Dspark.driver.port=43675'
-Dspark.yarn.app.container.log.dir=<LOG_DIR>
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
akka.tcp://sparkDriver@10.12.167.42:43675/user/CoarseGrainedScheduler
--executor-id 2 --hostname hadoop-Aspire-TC-606 --cores 1 --app-id
application_1452320323183_0007 --user-class-path file:$PWD/__app__.jar 1>
<LOG_DIR>/stdout 2> <LOG_DIR>/stderr
===============================================================================
16/01/11 15:12:37 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
hadoop-Aspire-TC-606:50804
16/01/11 15:12:40 INFO yarn.YarnAllocator: Completed container
container_1452320323183_0007_01_000003 (state: COMPLETE, exit status: 1)
16/01/11 15:12:40 INFO yarn.YarnAllocator: Container marked as failed:
container_1452320323183_0007_01_000003. Exit status: 1. Diagnostics: Exception
from container-launch.
Container id: container_1452320323183_0007_01_000003
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
