Hello,

First of all I m a newbie in Spark ,

I m trying to start the spark-shell with yarn cluster by running:

$ spark-shell --master yarn-client

Sometimes it goes well, but most of the time I got an error:

Container exited with a non-zero exit code 10
Failing this attempt. Failing the application.
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1437145851944
         final status: FAILED
         tracking URL:
http://My-HadoopServer:50080/cluster/app/application_1437033338028_0030
         user: hadoop
org.apache.spark.SparkException: Yarn application has already ended! It
might have been killed or unable to launch application master.
        at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:115)
....
....
....

searching in the yarn logs I got this log

$ yarn logs -applicationId application_1437033338028_0030
2015-07-17 17:11:03,961 - INFO
[sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3@74]
- Starting remoting
2015-07-17 17:11:04,200 - ERROR
[sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$1@65]
- failed to bind to My-HadoopServer/10.98.105.11:0, shutting down Netty
transport
2015-07-17 17:11:04,210 - WARN  [main:Logging$class@71] - Service
'sparkYarnAM' could not bind on port 0. Attempting port 1.
...
...
...
2015-07-17 17:11:05,123 - ERROR [main:Logging$class@96] - Uncaught
exception:
java.net.BindException: Failed to bind to:
My-HadoopServer/HadoopServerIP:0: Service 'sparkYarnAM' failed after 16
retries!
        at
org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
        at
akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:393)
        at
akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389)
        at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
...
....
....



I m using Spark 1.3, Hadoop 2.6 ,

 and in spark-env.sh it points to my hadoop configuration:

export HADOOP_CONF_DIR=/usr/hdp/2.2.4.4-16/hadoop/conf


Is this probleme coming from spark configuration or yarn configuration (or
spark with yarn confs)????

Any Ideas??



Amjad

Reply via email to