Hi Sofia,
Thank you for the detailed guide.
I confirmed that there are compatibility issues between Hive 1.2.1 and Spark
1.5.X. Hive 1.2.1 is by default built against spark 1.3.1 and compatible with
spark 1.4.1,but doesn't work with 1.5.X.
Basically,I am using somewhat the same configuration as yours,but I am using
yarn-client as the spark.master.
Did you try to use yarn-client as the spark.master?
I am building spark1.3.1 without hive and hive-thriftserver. I will get back to
you when I do the test,
Thank you, Sofia.
At 2016-01-11 23:56:32, "Sofia" <sofia.panagiot...@taiger.com> wrote:
From what I have seen you need to use a Spark version that does not contain the
Hive jars. (maybe build one as described at the page).
Maybe someone can respond to the compatibility issue better than me.
However, this is how I managed to make hive work with Spark
I had similar issues with the downloaded Spark 1.5.2 and in the end I used the
prebuilt Spark 1.4.1 with my Hive 1.2.1 (and Hadoop 2.7.1).
When doing the setup I used the configurations below:
hive-env.sh
export SPARK_HOME=/usr/local/spark
export HADOOP_HOME=/usr/local/hadoop
hive-site.xml
<property>
<name>hive.execution.engine</name>
<value>spark</value>
</property>
<property>
<name>spark.master</name>
<value>spark://spark-master:7077</value>
</property>
spark-env.sh
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export SPARK_HOME=/usr/local/spark
export SPARK_DIST_CLASSPATH=`hadoop classpath`
export SPARK_CLASSPATH=$CLASSPATH
I don’t know the answer to your issue exactly as at some point I faced the
same, but I hope this might help.
Cheers
On 11 Jan 2016, at 11:39, Todd <bit1...@163.com> wrote:
Thank you, Sofia.
From the log, looks it is java.lang.AbstractMethodError that leads to the job
failure.
I am using Hive 1.2.1+Spark 1.5.2, is this a compatibility issue?
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkBuildPlan
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkCreateTran.Map 1
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO Configuration.deprecation: mapred.task.is.map is deprecated.
Instead, use mapreduce.task.ismap
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO exec.Utilities: Processing alias t1
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO exec.Utilities: Adding input file
hdfs://hadoop.bit.com:9000/user/hive/warehouse/db_1.db/t1
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: <PERFLOG method=serializePlan
from=org.apache.hadoop.hive.ql.exec.Utilities>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO exec.Utilities: Serializing MapWork via kryo
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: </PERFLOG method=serializePlan
start=1452505326307 end=1452505326332 duration=25
from=org.apache.hadoop.hive.ql.exec.Utilities>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO storage.MemoryStore: ensureFreeSpace(505800) called with
curMem=0, maxMem=278302556
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory
(estimated size 493.9 KB, free 264.9 MB)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO storage.MemoryStore: ensureFreeSpace(42143) called with
curMem=505800, maxMem=278302556
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in
memory (estimated size 41.2 KB, free 264.9 MB)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on
10.12.167.42:38883 (size: 41.2 KB, free: 265.4 MB)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 ERROR util.Utils: uncaught error in thread SparkListenerBus, stopping
SparkContext
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl:
java.lang.AbstractMethodError
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:62)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:56)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:79)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1136)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63)
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO spark.SparkContext: Created broadcast 0 from hadoopRDD at
SparkPlanGenerator.java:188
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/metrics/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/api,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/static,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/threadDump,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/executors,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/environment/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/environment,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/rdd,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/storage,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/pool/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/pool,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/stage,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/stages,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/job/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/job,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs/json,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO handler.ContextHandler: stopped
o.s.j.s.ServletContextHandler{/jobs,null}
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: </PERFLOG method=SparkCreateTran.Map 1
start=1452505326274 end=1452505326824 duration=550
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkCreateTran.Reducer 2
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: <PERFLOG method=serializePlan
from=org.apache.hadoop.hive.ql.exec.Utilities>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO exec.Utilities: Serializing ReduceWork via kryo
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO ui.SparkUI: Stopped Spark web UI at http://10.12.167.42:4040
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO scheduler.DAGScheduler: Stopping DAGScheduler
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO cluster.YarnClientSchedulerBackend: Asking each executor to shut
down
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO cluster.YarnClientSchedulerBackend: Stopped
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: </PERFLOG method=serializePlan
start=1452505326849 end=1452505326874 duration=25
from=org.apache.hadoop.hive.ql.exec.Utilities>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: </PERFLOG method=SparkCreateTran.Reducer 2
start=1452505326824 end=1452505326936 duration=112
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: </PERFLOG method=SparkBuildPlan
start=1452505326273 end=1452505326936 duration=663
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator>
16/01/11 17:42:06 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:06 INFO log.PerfLogger: <PERFLOG method=SparkBuildRDDGraph
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlan>
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:07 INFO log.PerfLogger: </PERFLOG method=SparkBuildRDDGraph
start=1452505326936 end=1452505327000 duration=64
from=org.apache.hadoop.hive.ql.exec.spark.SparkPlan>
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: 16/01/11
17:42:07 INFO client.RemoteDriver: Failed to run job
b5a8886e-8229-4773-b943-b3c6f3c710bf
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl:
java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.SparkContext.org$apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.SparkContext.submitJob(SparkContext.scala:1981)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1.apply(AsyncRDDActions.scala:118)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.AsyncRDDActions$$anonfun$foreachAsync$1.apply(AsyncRDDActions.scala:116)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.rdd.AsyncRDDActions.foreachAsync(AsyncRDDActions.scala:116)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.api.java.JavaRDDLike$class.foreachAsync(JavaRDDLike.scala:690)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.spark.api.java.AbstractJavaRDDLike.foreachAsync(JavaRDDLike.scala:47)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:257)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:366)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:335)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
java.util.concurrent.FutureTask.run(FutureTask.java:262)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
16/01/11 17:42:07 [stderr-redir-1]: INFO client.SparkClientImpl: at
java.lang.Thread.run(Thread.java:745)
16/01/11 17:42:07 [RPC-Handler-3]: INFO client.SparkClientImpl: Received result
for b5a8886e-8229-4773-b943-b3c6f3c710bf
Status: Failed
16/01/11 17:42:07 [main]: ERROR status.SparkJobMonitor: Status: Failed
At 2016-01-11 17:17:51, "Sofia" <sofia.panagiot...@taiger.com> wrote:
Check log files to see the exact error. Alternatively run hive like this, to
get the exact error
hive --hiveconf hive.root.logger=INFO,console
There is a good chance that you are encountering problems between the Hive and
Spark versions and installation.
See
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
On 11 Jan 2016, at 08:47, Todd <bit1...@163.com> wrote:
Hive,
I am trying out the Hive on Spark with hive 1.2.1 and spark 1.5.2. Could
someone help me on this? Thanks!
Following are my steps:
1. build spark 1.5.2 without Hive and Hive Thrift Server. At this point, I can
use it to submit application using spark-submit --master yarn-client
2. And the built spark assembly jar into $HIVE_HOME/lib
3. start hive and add the following parameters
hive > set spark.master=yarn-client
hive > set spark.executor.memory=512M
hive > set spark.driver.memory=512M
hive > set spark.executor.instances=1
4. Then I run a simple query : select count(1) from t1;
The job fails will following error:
===============================================================================
YARN executor launch context:
env:
CLASSPATH ->
{{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
SPARK_LOG_URL_STDERR ->
http://hadoop-Aspire-TC-606:8042/node/containerlogs/container_1452320323183_0007_01_000003/hadoop/stderr?start=-4096
SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1452320323183_0007
SPARK_YARN_CACHE_FILES_FILE_SIZES -> 142746538
SPARK_USER -> hadoop
SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE
SPARK_YARN_MODE -> true
SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1452496343550
SPARK_LOG_URL_STDOUT ->
http://hadoop-Aspire-TC-606:8042/node/containerlogs/container_1452320323183_0007_01_000003/hadoop/stdout?start=-4096
SPARK_YARN_CACHE_FILES ->
hdfs://hadoop.bit.com:9000/user/hadoop/.sparkStaging/application_1452320323183_0007/spark-assembly-1.5.2-hadoop2.6.0.jar#__spark__.jar
command:
{{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms512m
-Xmx512m
'-Dhive.spark.log.dir=/home/hadoop/software/bigdata/spark-1.5.2-bin-hadoop2.6.0-withouthive/logs/'
-Djava.io.tmpdir={{PWD}}/tmp '-Dspark.driver.port=43675'
-Dspark.yarn.app.container.log.dir=<LOG_DIR>
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
akka.tcp://sparkDriver@10.12.167.42:43675/user/CoarseGrainedScheduler
--executor-id 2 --hostname hadoop-Aspire-TC-606 --cores 1 --app-id
application_1452320323183_0007 --user-class-path file:$PWD/__app__.jar 1>
<LOG_DIR>/stdout 2> <LOG_DIR>/stderr
===============================================================================
16/01/11 15:12:37 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
hadoop-Aspire-TC-606:50804
16/01/11 15:12:40 INFO yarn.YarnAllocator: Completed container
container_1452320323183_0007_01_000003 (state: COMPLETE, exit status: 1)
16/01/11 15:12:40 INFO yarn.YarnAllocator: Container marked as failed:
container_1452320323183_0007_01_000003. Exit status: 1. Diagnostics: Exception
from container-launch.
Container id: container_1452320323183_0007_01_000003
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
