Re: Execution blocked when collect()ing some relatively big blocks on spark 0.9

[Guillaume Pitel]

So, the problem occur when the size of the data is > 10MB, even though I've set the akka.frameSize to 512 (and it's showing this in the environment) I've had to set this to 512 on SPark 0.8.0, and I think it's not needed anymore since 0.8.1, so I'm going to go back to 10MB to see what's happening. But anyway, it's a bug if akka.frameSize cause problems. Guillaume runs till completion, but on occasion, it will just hang. The driver says, its sent all the tasks for the next stage. The executors, however, stop logging after sending back the results of the completed tasks from the previous stage. Do you know what's the size of the result sent back to the driver when it fails ? It should be logged -- Guillaume PITEL, Président +33(0)6 25 48 86 80 eXenSa S.A.S. 41, rue Périer - 92120 Montrouge - FRANCE Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05 -- Guillaume PITEL, Président +33(0)6 25 48 86 80 eXenSa S.A.S. 41, rue Périer - 92120 Montrouge - FRANCE Tel +33(0)1 84 16 36 77 / Fax +33(0)9 72 28 37 05