[
https://issues.apache.org/jira/browse/MESOS-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041895#comment-14041895
]
Ajay Viswanathan commented on MESOS-1535:
-----------------------------------------
I've searched all over for this issue and nothing seems to work. It surely
seems to be an issue with mesos, because the code runs as expected with Spark
standalone mode. I've tried this on both Spark 1.0.0 and Spark 0.9.1
Could somebody explain the error logs so that it may help me in finding the
issue?
> Pyspark on Mesos scheduler error
> --------------------------------
>
> Key: MESOS-1535
> URL: https://issues.apache.org/jira/browse/MESOS-1535
> Project: Mesos
> Issue Type: Bug
> Affects Versions: 0.18.0, 0.18.1
> Environment: Running a Mesos on a cluster of Centos 6.5 machines. 180
> GB memory.
> Reporter: Ajay Viswanathan
> Labels: pyspark
>
> This is an error that I get while running fine-grained PySpark on the mesos
> cluster. This comes after running some 200-1000 tasks generally.
> Pyspark code:
> while True:
> sc.parallelize(range(10)).map(lambda n : n*2).collect()
> Error log:
> org.apache.spark.SparkException: EOF reached before Python server acknowledged
> at
> org.apache.spark.api.python.PythonAccumulatorParam.addInPlace(PythonR
>
> DD.scala:322)
> at
> org.apache.spark.api.python.PythonAccumulatorParam.addInPlace(PythonR
>
> DD.scala:293)
> at org.apache.spark.Accumulable.$plus$plus$eq(Accumulators.scala:70)
> at
> org.apache.spark.Accumulators$$anonfun$add$2.apply(Accumulators.scala
>
> :275)
> at
> org.apache.spark.Accumulators$$anonfun$add$2.apply(Accumulators.scala
>
> :273)
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(
>
> TraversableLike.scala:772)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.sca
>
> la:98)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.sca
>
> la:98)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala
>
> :226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.s
>
> cala:771)
> at org.apache.spark.Accumulators$.add(Accumulators.scala:273)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGSched
>
> uler.scala:826)
> at
> org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.sca
>
> la:601)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$ano
>
>
> nfun$receive$1.applyOrElse(DAGScheduler.scala:190)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
> at akka.actor.ActorCell.invoke(ActorCell.scala:456)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(Abst
>
> ractDispatcher.scala:386)
> at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool
>
> .java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:19
>
> 79)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThre
>
> ad.java:107)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
