Get attempt number in a closure

2014-10-20 Thread Yin Huai
Hello, Is there any way to get the attempt number in a closure? Seems TaskContext.attemptId actually returns the taskId of a task (see this https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/Executor.scala#L181 and this https://github.com/apache/spark/blob

Re: Get attempt number in a closure

2014-10-20 Thread Reynold Xin
wrote: Hello, Is there any way to get the attempt number in a closure? Seems TaskContext.attemptId actually returns the taskId of a task (see this https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/Executor.scala#L181 and this https://github.com

Re: Get attempt number in a closure

2014-10-20 Thread Yin Huai
think part of the problem is that we don't actually have the attempt id on the executors. If we do, that's great. If not, we'd need to propagate that over. On Mon, Oct 20, 2014 at 7:17 AM, Yin Huai huaiyin@gmail.com wrote: Hello, Is there any way to get the attempt number in a closure

Re: Get attempt number in a closure

2014-10-20 Thread Patrick Wendell
on the executors. If we do, that's great. If not, we'd need to propagate that over. On Mon, Oct 20, 2014 at 7:17 AM, Yin Huai huaiyin@gmail.com wrote: Hello, Is there any way to get the attempt number in a closure? Seems TaskContext.attemptId actually returns the taskId of a task (see

Re: Get attempt number in a closure

2014-10-20 Thread Kay Ousterhout
, Yin Huai huaiyin@gmail.com wrote: Hello, Is there any way to get the attempt number in a closure? Seems TaskContext.attemptId actually returns the taskId of a task (see this https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/Executor.scala

Re: Get attempt number in a closure

2014-10-20 Thread Kay Ousterhout
great. If not, we'd need to propagate that over. On Mon, Oct 20, 2014 at 7:17 AM, Yin Huai huaiyin@gmail.com wrote: Hello, Is there any way to get the attempt number in a closure? Seems TaskContext.attemptId actually returns the taskId of a task (see this https

Re: Get attempt number in a closure

2014-10-20 Thread Reynold Xin
need to propagate that over. On Mon, Oct 20, 2014 at 7:17 AM, Yin Huai huaiyin@gmail.com wrote: Hello, Is there any way to get the attempt number in a closure? Seems TaskContext.attemptId actually returns the taskId of a task (see this https://github.com/apache/spark/blob

Re: Get attempt number in a closure

2014-10-20 Thread Yin Huai
to get the attempt number in a closure? Seems TaskContext.attemptId actually returns the taskId of a task (see this https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/Executor.scala#L181 and this https://github.com/apache/spark/blob/master