From what I have seen you need to use a Spark version that does not contain the
Hive jars. (maybe build one as described at the page).
Maybe someone can respond to the compatibility issue better than me.
However, this is how I managed to make hive work with Spark
I had similar issues with the downloaded Spark 1.5.2 and in the end I used the
prebuilt Spark 1.4.1 with my Hive 1.2.1 (and Hadoop 2.7.1).
When doing the setup I used the configurations below:
hive-env.sh
export SPARK_HOME=/usr/local/spark
export HADOOP_HOME=/usr/local/hadoop
hive-site.xml
<property>
<name>hive.execution.engine</name>
<value>spark</value>
</property>
<property>
<name>spark.master</name>
<value>spark://spark-master:7077</value>
</property>
spark-env.sh
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export SPARK_HOME=/usr/local/spark
export SPARK_DIST_CLASSPATH=`hadoop classpath`
export SPARK_CLASSPATH=$CLASSPATH
I don’t know the answer to your issue exactly as at some point I faced the
same, but I hope this might help.
Cheers
> On 11 Jan 2016, at 11:39, Todd <bit1...@163.com> wrote:
>
>
> Thank you, Sofia.
>
> From the log, looks it is java.lang.AbstractMethodError that leads to the job
> failure.
> I am using Hive 1.2.1+Spark 1.5.2, is this a compatibility issue?
>
>
>
>
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkBuildPlan
> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkCreateTran.Map 1
> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO Configuration.deprecation: mapred.task.is.map is deprecated.
> Instead, use mapreduce.task.ismap
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO exec.Utilities: Processing alias t1
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO exec.Utilities: Adding input file
> hdfs://hadoop.bit.com:9000/user/hive/warehouse/db_1.db/t1
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO log.PerfLogger: <PERFLOG method=serializePlan
> from=org.apache.hadoop.hive.ql.exec.Utilities>
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO exec.Utilities: Serializing MapWork via kryo
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO log.PerfLogger: </PERFLOG method=serializePlan
> start=1452505326307 end=1452505326332 duration=25
> from=org.apache.hadoop.hive.ql.exec.Utilities>
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO storage.MemoryStore: ensureFreeSpace(505800) called with
> curMem=0, maxMem=278302556
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO storage.MemoryStore: Block broadcast_0 stored as values in
> memory (estimated size 493.9 KB, free 264.9 MB)
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO storage.MemoryStore: ensureFreeSpace(42143) called with
> curMem=505800, maxMem=278302556
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes
> in memory (estimated size 41.2 KB, free 264.9 MB)
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on
> 10.12.167.42:38883 (size: 41.2 KB, free: 265.4 MB)
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 ERROR util.Utils: uncaught error in thread SparkListenerBus,
> stopping SparkContext
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl:
> java.lang.AbstractMethodError
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:62)
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56)
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37)
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79)
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136)
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO spark.SparkContext: Created broadcast 0 from hadoopRDD at
> SparkPlanGenerator.java:188
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/metrics/json,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/api,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/static,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/executors/threadDump,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/executors/json,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/executors,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/environment/json,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/environment,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/storage/rdd,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/storage/json,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/storage,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/stages/pool/json,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/stages/pool,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/stages/stage/json,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/stages/stage,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/stages/json,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/stages,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/jobs/job/json,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/jobs/job,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/jobs/json,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO handler.ContextHandler: stopped
> o.s.j.s.ServletContextHandler{/jobs,null}
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO log.PerfLogger: </PERFLOG method=SparkCreateTran.Map 1
> start=1452505326274 end=1452505326824 duration=550
> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkCreateTran.Reducer 2
> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO log.PerfLogger: <PERFLOG method=serializePlan
> from=org.apache.hadoop.hive.ql.exec.Utilities>
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO exec.Utilities: Serializing ReduceWork via kryo
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO ui.SparkUI: Stopped Spark web UI at http://10.12.167.42:4040
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO scheduler.DAGScheduler: Stopping DAGScheduler
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO cluster.YarnClientSchedulerBackend: Asking each executor to
> shut down
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO cluster.YarnClientSchedulerBackend: Stopped
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO log.PerfLogger: </PERFLOG method=serializePlan
> start=1452505326849 end=1452505326874 duration=25
> from=org.apache.hadoop.hive.ql.exec.Utilities>
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO log.PerfLogger: </PERFLOG method=SparkCreateTran.Reducer 2
> start=1452505326824 end=1452505326936 duration=112
> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO log.PerfLogger: </PERFLOG method=SparkBuildPlan
> start=1452505326273 end=1452505326936 duration=663
> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
> 16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkBuildRDDGraph
> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlan>
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:07 INFO log.PerfLogger: </PERFLOG method=SparkBuildRDDGraph
> start=1452505326936 end=1452505327000 duration=64
> from=org.apache.hadoop.hive.ql.exec.spark.SparkPlan>
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
> 17:42:07 INFO client.RemoteDriver: Failed to run job
> b5a8886e-8229-4773-b943-b3c6f3c710bf
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl:
> java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.SparkContext.org$apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.SparkContext.submitJob(SparkContext.scala:1981)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1.apply(AsyncRDDActions.scala:118)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1.apply(AsyncRDDActions.scala:116)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.rdd.AsyncRDDActions.foreachAsync(AsyncRDDActions.scala:116)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.api.java.JavaRDDLike$class.foreachAsync(JavaRDDLike.scala:690)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.spark.api.java.AbstractJavaRDDLike.foreachAsync(JavaRDDLike.scala:47)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:257)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:366)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:335)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
> java.lang.Thread.run(Thread.java:745)
> 16/01/11 17:42:07 [RPC-Handler-3]: INFO client.SparkClientImpl: Received
> result for b5a8886e-8229-4773-b943-b3c6f3c710bf
> Status: Failed
> 16/01/11 17:42:07 [main]: ERROR status.SparkJobMonitor: Status: Failed
>
>
>
>
> At 2016-01-11 17:17:51, "Sofia" <sofia.panagiot...@taiger.com> wrote:
> Check log files to see the exact error. Alternatively run hive like this, to
> get the exact error
>
> hive --hiveconf hive.root.logger=INFO,console
>
> There is a good chance that you are encountering problems between the Hive
> and Spark versions and installation.
> See
> https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
>
> <https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started>
>
>
>> On 11 Jan 2016, at 08:47, Todd <bit1...@163.com <mailto:bit1...@163.com>>
>> wrote:
>>
>> Hive,
>>
>> I am trying out the Hive on Spark with hive 1.2.1 and spark 1.5.2. Could
>> someone help me on this? Thanks!
>>
>> Following are my steps:
>> 1. build spark 1.5.2 without Hive and Hive Thrift Server. At this point, I
>> can use it to submit application using spark-submit --master yarn-client
>> 2. And the built spark assembly jar into $HIVE_HOME/lib
>> 3. start hive and add the following parameters
>> hive > set spark.master=yarn-client
>> hive > set spark.executor.memory=512M
>> hive > set spark.driver.memory=512M
>> hive > set spark.executor.instances=1
>>
>> 4. Then I run a simple query : select count(1) from t1;
>>
>> The job fails will following error:
>>
>> ===============================================================================
>> YARN executor launch context:
>> env:
>> CLASSPATH ->
>> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
>> SPARK_LOG_URL_STDERR ->
>> http://hadoop-Aspire-TC-606:8042/node/containerlogs/container_1452320323183_0007_01_000003/hadoop/stderr?start=-4096
>>
>> <http://hadoop-aspire-tc-606:8042/node/containerlogs/container_1452320323183_0007_01_000003/hadoop/stderr?start=-4096>
>> SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1452320323183_0007
>> SPARK_YARN_CACHE_FILES_FILE_SIZES -> 142746538
>> SPARK_USER -> hadoop
>> SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE
>> SPARK_YARN_MODE -> true
>> SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1452496343550
>> SPARK_LOG_URL_STDOUT ->
>> http://hadoop-Aspire-TC-606:8042/node/containerlogs/container_1452320323183_0007_01_000003/hadoop/stdout?start=-4096
>>
>> <http://hadoop-aspire-tc-606:8042/node/containerlogs/container_1452320323183_0007_01_000003/hadoop/stdout?start=-4096>
>> SPARK_YARN_CACHE_FILES ->
>> hdfs://hadoop.bit.com:9000/user/hadoop/.sparkStaging/application_1452320323183_0007/spark-assembly-1.5.2-hadoop2.6.0.jar#__spark__.jar
>>
>> <hdfs://hadoop.bit.com:9000/user/hadoop/.sparkStaging/application_1452320323183_0007/spark-assembly-1.5.2-hadoop2.6.0.jar#__spark__.jar>
>>
>> command:
>> {{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms512m
>> -Xmx512m
>> '-Dhive.spark.log.dir=/home/hadoop/software/bigdata/spark-1.5.2-bin-hadoop2.6.0-withouthive/logs/'
>> -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.driver.port=43675'
>> -Dspark.yarn.app.container.log.dir=<LOG_DIR>
>> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
>> akka.tcp://sparkDriver@10.12.167.42:43675/user/CoarseGrainedScheduler
>> <akka.tcp://sparkDriver@10.12.167.42:43675/user/CoarseGrainedScheduler>
>> --executor-id 2 --hostname hadoop-Aspire-TC-606 --cores 1 --app-id
>> application_1452320323183_0007 --user-class-path file:$PWD/__app__.jar 1>
>> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr
>> ===============================================================================
>>
>> 16/01/11 15:12:37 INFO impl.ContainerManagementProtocolProxy: Opening proxy
>> : hadoop-Aspire-TC-606:50804
>> 16/01/11 15:12:40 INFO yarn.YarnAllocator: Completed container
>> container_1452320323183_0007_01_000003 (state: COMPLETE, exit status: 1)
>> 16/01/11 15:12:40 INFO yarn.YarnAllocator: Container marked as failed:
>> container_1452320323183_0007_01_000003. Exit status: 1. Diagnostics:
>> Exception from container-launch.
>> Container id: container_1452320323183_0007_01_000003
>> Exit code: 1
>> Stack trace: ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> at
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> Container exited with a non-zero exit code 1
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
