Hi,
I've been trying to resolve this issue for the past couple of weeks without
any success. I had given up running a standalone app because I was able to
use the REPL for most of my work but now I really need to run an app. This
is becoming a potential showstopper for me in using Spark for my use case.
My application is very simple. It reads a file from HDFS and does a count
on the RDD.
I believe my cluster is setup correctly because I can connect to it from my
REPL and I can see all the worker nodes on the Spark UI. Is there any other
test or way of figuring out if there is an issue my cluster setup. I'll try
to provide as much detail as possible before. I've asked this question on
the list before but I'm asking it again with more relevant details (all in
one place).
Here is the exception.
*14/02/16 21:45:59 WARN scheduler.TaskSchedulerImpl: Initial job has not
accepted any resources; check your cluster UI to ensure that workers are
registered and have sufficient memory*
14/02/16 21:46:13 ERROR client.AppClient$ClientActor: All masters are
unresponsive! Giving up.
14/02/16 21:46:13 ERROR cluster.SparkDeploySchedulerBackend: Spark cluster
looks dead, giving up.
14/02/16 21:46:13 INFO scheduler.TaskSchedulerImpl: Remove TaskSet 0.0 from
pool
*14/02/16 21:46:13 INFO scheduler.DAGScheduler: Failed to run count at
SimpleApp.scala:24*
[error] (run-main) org.apache.spark.SparkException: Job aborted: Spark
cluster looks down
org.apache.spark.SparkException: Job aborted: Spark cluster looks down
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1028)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1026)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1026)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:619)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:207)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
*My directory structure*
$find .
.
./sbt
./simple.sbt
./sbt-launch-0.12.4.jar
./src
./src/main
./src/main/scala
./src/main/scala/SimpleApp.scala
*Simple.sbt *
name := "Simple Project"
version := "1.0"
scalaVersion := "2.10.0"
*libraryDependencies += "org.apache.spark" % "spark-core_2.10" %
"0.9.0-incubating_SNAPSHOT" //This lib is build on my local version of
Spark and taken from my ~/.m2/repository *
libraryDependencies += "org.apache.hadoop" % "hadoop-client" %
"0.20.2-cdh3u6"
resolvers ++= Seq("Akka Repository" at "http://repo.akka.io/releases/","Spray
Repository" at "http://repo.spray.cc/";, *"Local Maven Repository" at
"file://"+Path.userHome.absolutePath+"/.m2/repository"*)
Simple.scala
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
object SimpleApp{
def main(args: Array[String]){
val conf = new SparkConf()
.setMaster("*spark://aa.bb.xxx.yy:7077*")
.setAppName("Simple Analysis")
.set("spark.executor.memory", "4g")
val sc = new SparkContext(conf)
println(" -- Starting analysis --")
val textFile =
sc.textFile("hdfs://xxx.xxx.xxx.xxx:54310/tmp/data/error.log")
println(" -- Done reading file from HDFS --")
textFile.cache
println(*textFile.count*)
}
}
Please note that Spark master URL above - *spark://aa.bb.xxx.yy:7077* is
exactly the same that shows up on my Spark Web UI.
Also, I use the following to connect to using the Spark shell and it works
fine. I can also run the same code above on the REPL without any
exceptions.
MASTER="*spark://aa.bb.xxx.yy:7077*" SPARK_MEM="24g" ./spark-shell
Any help to resolve this would be greatly appreciated and very helpful.
Thanks
-Soumya
