Re: KafkaDirectStream can't be recovered from checkpoint

2015-09-21 Thread Petr Novak
We have tried on another cluster installation with the same effect. Petr On Mon, Sep 21, 2015 at 10:45 AM, Petr Novak wrote: > It might be connected with my problems with gracefulShutdown in Spark >> 1.5.0 2.11 >> https://mail.google.com/mail/#search/petr/14fb6bd5166f9395 >> >> Maybe Ctrl+C cor

Re: KafkaDirectStream can't be recovered from checkpoint

2015-09-21 Thread Petr Novak
> > It might be connected with my problems with gracefulShutdown in Spark > 1.5.0 2.11 > https://mail.google.com/mail/#search/petr/14fb6bd5166f9395 > > Maybe Ctrl+C corrupts checkpoints and breaks gracefulShutdown? > The provided link is obviously wrong. I haven't found it Spark mailing lists archi

Re: KafkaDirectStream can't be recovered from checkpoint

2015-09-20 Thread Petr Novak
Hi Michal, yes, it is there logged twice, it can be seen in attached log in one of previous post with more details: 15/09/17 23:06:37 INFO StreamingContext: Invoking stop(stopGracefully=false) from shutdown hook 15/09/17 23:06:37 INFO StreamingContext: Invoking stop(stopGracefully=false) from shut

Re: KafkaDirectStream can't be recovered from checkpoint

2015-09-18 Thread Michal Čizmazia
Hi Petr, after Ctrl+C can you see the following message in the logs? Invoking stop(stopGracefully=false) Details: https://github.com/apache/spark/pull/6307 On 18 September 2015 at 10:28, Petr Novak wrote: > It might be connected with my problems with gracefulShutdown in Spark > 1.5.0 2.11 > h

Re: KafkaDirectStream can't be recovered from checkpoint

2015-09-18 Thread Petr Novak
It might be connected with my problems with gracefulShutdown in Spark 1.5.0 2.11 https://mail.google.com/mail/#search/petr/14fb6bd5166f9395 Maybe Ctrl+C corrupts checkpoints and breaks gracefulShutdown? Petr On Fri, Sep 18, 2015 at 4:10 PM, Petr Novak wrote: > ...to ensure it is not something

Re: KafkaDirectStream can't be recovered from checkpoint

2015-09-18 Thread Petr Novak
...to ensure it is not something wrong on my cluster. On Fri, Sep 18, 2015 at 4:09 PM, Petr Novak wrote: > I have tried it on Spark 1.3.0 2.10 and it works. The same code doesn't on > Spark 1.5.0 2.11. It would be nice if anybody could try on another > installation to ensure it is something wron

Re: KafkaDirectStream can't be recovered from checkpoint

2015-09-18 Thread Petr Novak
I have tried it on Spark 1.3.0 2.10 and it works. The same code doesn't on Spark 1.5.0 2.11. It would be nice if anybody could try on another installation to ensure it is something wrong on my cluster. Many thanks, Petr On Fri, Sep 18, 2015 at 4:07 PM, Petr Novak wrote: > This one is generated,

Re: KafkaDirectStream can't be recovered from checkpoint

2015-09-18 Thread Petr Novak
This one is generated, I suppose, after Ctrl+C 15/09/18 14:38:25 INFO Worker: Asked to kill executor app-20150918143823-0001/0 15/09/18 14:38:25 INFO Worker: Asked to kill executor app-20150918143823-0001/0 15/09/18 14:38:25 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handle

Re: KafkaDirectStream can't be recovered from checkpoint

2015-09-18 Thread Tathagata Das
Also, giving more log4j messages around the error area would be useful. On Thu, Sep 17, 2015 at 1:02 PM, Cody Koeninger wrote: > Is there a particular reason you're calling checkpoint on the stream in > addition to the streaming context? > > On Thu, Sep 17, 2015 at 2:36 PM, Petr Novak wrote: >

Re: KafkaDirectStream can't be recovered from checkpoint

2015-09-17 Thread Cody Koeninger
Is there a particular reason you're calling checkpoint on the stream in addition to the streaming context? On Thu, Sep 17, 2015 at 2:36 PM, Petr Novak wrote: > Hi all, > it throws FileBasedWriteAheadLogReader: Error reading next item, EOF > reached > java.io.EOFException > at java.io.DataInput