The data is small. The job is composed of many small stages.
* I found that with fewer than 222 the problem exhibits. What will be
gained by going higher?
* Pushing up the parallelism only pushes up the boundary at which the
system appears to hang. I'm worried about some sort of message loss or
inconsistency.
* Yes, we are using Kryo.
* I'll try that, but I'm again a little confused why you're recommending
this. I'm stumped so might as well?
On Wed, Feb 25, 2015 at 11:13 PM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> What operation are you trying to do and how big is the data that you are
> operating on?
>
> Here's a few things which you can try:
>
> - Repartition the RDD to a higher number than 222
> - Specify the master as local[*] or local[10]
> - Use Kryo Serializer (.set("spark.serializer",
> "org.apache.spark.serializer.KryoSerializer"))
> - Enable RDD Compression (.set("spark.rdd.compress","true") )
>
>
> Thanks
> Best Regards
>
> On Thu, Feb 26, 2015 at 10:15 AM, Victor Tso-Guillen <v...@paxata.com>
> wrote:
>
>> I'm getting this really reliably on Spark 1.2.1. Basically I'm in local
>> mode with parallelism at 8. I have 222 tasks and I never seem to get far
>> past 40. Usually in the 20s to 30s it will just hang. The last logging is
>> below, and a screenshot of the UI.
>>
>> 2015-02-25 20:39:55.779 GMT-0800 INFO [task-result-getter-3]
>> TaskSetManager - Finished task 3.0 in stage 16.0 (TID 22) in 612 ms on
>> localhost (1/5)
>> 2015-02-25 20:39:55.825 GMT-0800 INFO [Executor task launch worker-10]
>> Executor - Finished task 1.0 in stage 16.0 (TID 20). 2492 bytes result sent
>> to driver
>> 2015-02-25 20:39:55.825 GMT-0800 INFO [Executor task launch worker-8]
>> Executor - Finished task 2.0 in stage 16.0 (TID 21). 2492 bytes result sent
>> to driver
>> 2015-02-25 20:39:55.831 GMT-0800 INFO [task-result-getter-0]
>> TaskSetManager - Finished task 1.0 in stage 16.0 (TID 20) in 670 ms on
>> localhost (2/5)
>> 2015-02-25 20:39:55.836 GMT-0800 INFO [task-result-getter-1]
>> TaskSetManager - Finished task 2.0 in stage 16.0 (TID 21) in 674 ms on
>> localhost (3/5)
>> 2015-02-25 20:39:55.891 GMT-0800 INFO [Executor task launch worker-9]
>> Executor - Finished task 0.0 in stage 16.0 (TID 19). 2492 bytes result sent
>> to driver
>> 2015-02-25 20:39:55.896 GMT-0800 INFO [task-result-getter-2]
>> TaskSetManager - Finished task 0.0 in stage 16.0 (TID 19) in 740 ms on
>> localhost (4/5)
>>
>> [image: Inline image 1]
>> What should I make of this? Where do I start?
>>
>> Thanks,
>> Victor
>>
>
>
