Hello, First of all I m a newbie in Spark ,
I m trying to start the spark-shell with yarn cluster by running: $ spark-shell --master yarn-client Sometimes it goes well, but most of the time I got an error: Container exited with a non-zero exit code 10 Failing this attempt. Failing the application. ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1437145851944 final status: FAILED tracking URL: http://My-HadoopServer:50080/cluster/app/application_1437033338028_0030 user: hadoop org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:115) .... .... .... searching in the yarn logs I got this log $ yarn logs -applicationId application_1437033338028_0030 2015-07-17 17:11:03,961 - INFO [sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3@74] - Starting remoting 2015-07-17 17:11:04,200 - ERROR [sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$1@65] - failed to bind to My-HadoopServer/10.98.105.11:0, shutting down Netty transport 2015-07-17 17:11:04,210 - WARN [main:Logging$class@71] - Service 'sparkYarnAM' could not bind on port 0. Attempting port 1. ... ... ... 2015-07-17 17:11:05,123 - ERROR [main:Logging$class@96] - Uncaught exception: java.net.BindException: Failed to bind to: My-HadoopServer/HadoopServerIP:0: Service 'sparkYarnAM' failed after 16 retries! at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272) at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:393) at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389) at scala.util.Success$$anonfun$map$1.apply(Try.scala:206) ... .... .... I m using Spark 1.3, Hadoop 2.6 , and in spark-env.sh it points to my hadoop configuration: export HADOOP_CONF_DIR=/usr/hdp/2.2.4.4-16/hadoop/conf Is this probleme coming from spark configuration or yarn configuration (or spark with yarn confs)???? Any Ideas?? Amjad