Re: Spark Job Execution halts during shuffle...

Hi Can someone throw light on this. The issue is not frquently happening. Sometimes the job halts with the above messages. Regards, Padma Ch On Fri, May 27, 2016 at 8:47 AM, Ted Yu wrote: > Priya: > Have you checked the executor logs on hostname1 and hostname2 ? > > Cheers

Re: Spark Job Execution halts during shuffle...

Priya: Have you checked the executor logs on hostname1 and hostname2 ? Cheers On Thu, May 26, 2016 at 8:00 PM, Takeshi Yamamuro wrote: > Hi, > > If you get stuck in job fails, one of best practices is to increase > #partitions. > Also, you'd better off using DataFrame

Re: Spark Job Execution halts during shuffle...

Hi, If you get stuck in job fails, one of best practices is to increase #partitions. Also, you'd better off using DataFrame instread of RDD in terms of join optimization. // maropu On Thu, May 26, 2016 at 11:40 PM, Priya Ch wrote: > Hello Team, > > > I am

Spark Job Execution halts during shuffle...

Hello Team, I am trying to perform join 2 rdds where one is of size 800 MB and the other is 190 MB. During the join step, my job halts and I don't see progress in the execution. This is the message I see on console - INFO spark.MapOutputTrackerMasterEndPoint: Asked to send map output