And when I use sparksubmit script, I get the following error:
py4j.protocol.Py4JJavaError: An error occurred while calling
o26.trainKMeansModel.
: org.apache.spark.SparkException: Job aborted due to stage failure: All
masters are unresponsive! Giving up.
at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1049)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1033)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1031)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1031)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:635)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:635)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:635)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1234)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
My spark submit code is
conf = SparkConf().set("spark.executor.memory",
"32G").set("spark.akka.frameSize", "1000")
sc = SparkContext(conf = conf)
rdd = sc.parallelize(matrix,5)
from pyspark.mllib.clustering import KMeans
from math import sqrt
clusters = KMeans.train(rdd, 5, maxIterations=2,runs=2,
initializationMode="random")
def error(point):
center = clusters.centers[clusters.predict(point)]
return sqrt(sum([x**2 for x in (point - center)]))
WSSSE = rdd.map(lambda point: error(point)).reduce(lambda x, y: x + y)
print "Within Set Sum of Squared Error = " + str(WSSSE)
Which is executed as following:
spark-submit --master $SPARKURL clustering_example.py --executor-memory
32G --driver-memory 60G
On Sun, Sep 14, 2014 at 10:47 AM, Chengi Liu <chengi.liu...@gmail.com>
wrote:
> How? Example please..
> Also, if I am running this in pyspark shell.. how do i configure
> spark.akka.frameSize ??
>
>
> On Sun, Sep 14, 2014 at 7:43 AM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> When the data size is huge, you better of use the torrentBroadcastFactory.
>>
>> Thanks
>> Best Regards
>>
>> On Sun, Sep 14, 2014 at 2:54 PM, Chengi Liu <chengi.liu...@gmail.com>
>> wrote:
>>
>>> Specifically the error I see when I try to operate on rdd created by
>>> sc.parallelize method
>>> : org.apache.spark.SparkException: Job aborted due to stage failure:
>>> Serialized task 12:12 was 12062263 bytes which exceeds spark.akka.frameSize
>>> (10485760 bytes). Consider using broadcast variables for large values.
>>>
>>> On Sun, Sep 14, 2014 at 2:20 AM, Chengi Liu <chengi.liu...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> I am trying to create an rdd out of large matrix.... sc.parallelize
>>>> suggest to use broadcast
>>>> But when I do
>>>>
>>>> sc.broadcast(data)
>>>> I get this error:
>>>>
>>>> Traceback (most recent call last):
>>>> File "<stdin>", line 1, in <module>
>>>> File "/usr/common/usg/spark/1.0.2/python/pyspark/context.py", line
>>>> 370, in broadcast
>>>> pickled = pickleSer.dumps(value)
>>>> File "/usr/common/usg/spark/1.0.2/python/pyspark/serializers.py",
>>>> line 279, in dumps
>>>> def dumps(self, obj): return cPickle.dumps(obj, 2)
>>>> SystemError: error return without exception set
>>>> Help?
>>>>
>>>>
>>>
>>
>
