Hi
I have installed a standalone Spark set up in standalone mode in a Linux
server and I am trying to access that spark setup from Java in windows. When
I try connecting to Spark I see the following exception
14/12/16 12:52:52 WARN TaskSchedulerImpl: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and
have sufficient memory
14/12/16 12:52:56 INFO AppClient$ClientActor: Connecting to master
spark://01hw294954.INDIA:7077...
14/12/16 12:53:07 WARN TaskSchedulerImpl: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and
have sufficient memory
14/12/16 12:53:16 INFO AppClient$ClientActor: Connecting to master
spark://01hw294954.INDIA:7077...
14/12/16 12:53:22 WARN TaskSchedulerImpl: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and
have sufficient memory
14/12/16 12:53:36 ERROR SparkDeploySchedulerBackend: Application has been
killed. Reason: All masters are unresponsive! Giving up.
14/12/16 12:53:36 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks
have all completed, from pool
14/12/16 12:53:36 INFO TaskSchedulerImpl: Cancelling stage 0
14/12/16 12:53:36 INFO DAGScheduler: Failed to run collect at
MySqlConnector.java:579
Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: All masters are unresponsive! Giving up.
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1033)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1017)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1015)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1015)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:633)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1207)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run
I have attached the Spark Master UI
Spark Master at spark://01hw294954.INDIA:7077
URL: spark://01hw294954.INDIA:7077
Workers: 1
Cores: 2 Total, 0 Used
Memory: 835.0 MB Total, 0.0 B Used
Applications: 0 Running, 0 Completed
Drivers: 0 Running, 0 Completed
Status: ALIVE
Workers
Id Address State Cores Memory
worker-20141216123503-01hw294954.INDIA-38962 01hw294954.INDIA:38962 ALIVE
2
(0 Used) 835.0 MB (0.0 B Used)
Running Applications
ID Name Cores Memory per Node Submitted Time User State Duration
Completed Applications
ID Name Cores Memory per Node Submitted Time User State Duration
My Spark Slave is
Spark Worker at 01hw294954.INDIA:38962
ID: worker-20141216123503-01hw294954.INDIA-38962
Master URL: spark://01hw294954.INDIA:7077
Cores: 2 (0 Used)
Memory: 835.0 MB (0.0 B Used)
Back to Master
Running Executors (0)
ExecutorID Cores State Memory Job Details Logs
My Java Master Code looks like this
SparkConf sparkConf = new SparkConf().setAppName("JdbcRddTest");
sparkConf.setMaster("spark://01hw294954.INDIA:7077");
When I tried using the same code with the local spark set up as the master
it ran.
Any help for solving this issue is very much appreciated.
Thanks and Regards
Jai
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Accessing-Apache-Spark-from-Java-tp20700.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
