Re: How does Spark set task indexes?

Yes I've noticed this one and its related cousin, but not sure this is the same issue there; our job "properly" ends after 6 attempts. We'll try with disabled speculative mode anyway! On 25 May 2016 at 00:13, Ted Yu wrote: > Have you taken a look at SPARK-14915 ? > > On

Re: How does Spark set task indexes?

Have you taken a look at SPARK-14915 ? On Tue, May 24, 2016 at 1:00 PM, Adrien Mogenet < adrien.moge...@contentsquare.com> wrote: > Hi, > > I'm wondering how Spark is setting the "index" of task? > I'm asking this question because we have a job that constantly fails at > task index = 421. > >

How does Spark set task indexes?

Hi, I'm wondering how Spark is setting the "index" of task? I'm asking this question because we have a job that constantly fails at task index = 421. When increasing number of partitions, this then fails at index=4421. Increase it a little bit more, now it's 24421. Our job is as simple as "(1)