Can you add sc.stop at the end of the code and try? On 1 Dec 2016 18:03, "Daniel van der Ende" <daniel.vandere...@gmail.com> wrote:
> Hi, > > I've seen this a few times too. Usually it indicates that your driver > doesn't have enough resources to process the result. Sometimes increasing > driver memory is enough (yarn memory overhead can also help). Is there any > specific reason for you to run in client mode and not in cluster mode? > Having run into this a number of times (and wanting to spare the resources > of our submitting machines) we have now switched to use yarn cluster mode > by default. This seems to resolve the problem. > > Hope this helps, > > Daniel > > On 29 Nov 2016 11:20 p.m., "Selvam Raman" <sel...@gmail.com> wrote: > >> Hi, >> >> I have submitted spark job in yarn client mode. The executor and cores >> were dynamically allocated. In the job i have 20 partitions, so 5 container >> each with 4 core has been submitted. It almost processed all the records >> but it never exit the job and in the application master container i am >> seeing the below error message. >> >> INFO yarn.YarnAllocator: Canceling requests for 0 executor containers >> WARN yarn.YarnAllocator: Expected to find pending requests, but found none. >> >> >> >> The same job i ran it for only 1000 records which successfully finished. >> >> >> Can anyone help me to sort out this issue. >> >> Spark version:2.0( AWS EMR). >> >> -- >> Selvam Raman >> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து" >> >
