Re: Spark driver and yarn behavior

On 20 May 2016, at 00:34, Shankar Venkataraman > wrote: Thanks Luciano. The case we are seeing is different - the yarn resource manager is shutting down the container in which the executor is running since there does

Re: Spark driver and yarn behavior

Thanks Luciano. The case we are seeing is different - the yarn resource manager is shutting down the container in which the executor is running since there does not seem to be a response and it is deeming it dead. It started another container but the driver seems to be oblivious for nearly 2

Re: Spark driver and yarn behavior

On Thu, May 19, 2016 at 3:16 PM, Shankar Venkataraman < shankarvenkataraman...@gmail.com> wrote: > Hi! > > We are running into an interesting behavior with the Spark driver. We > Spark running under Yarn. The spark driver seems to be sending work to a > dead executor for 3 hours before it

Spark driver and yarn behavior

Hi! We are running into an interesting behavior with the Spark driver. We Spark running under Yarn. The spark driver seems to be sending work to a dead executor for 3 hours before it recognizes it. The workload seems to have been processed by other executors just fine and we see no loss in