Re: KafkaDirectStream can't be recovered from checkpoint

> > It might be connected with my problems with gracefulShutdown in Spark > 1.5.0 2.11 > https://mail.google.com/mail/#search/petr/14fb6bd5166f9395 > > Maybe Ctrl+C corrupts checkpoints and breaks gracefulShutdown? > The provided link is obviously wrong. I haven't found it Spark mailing lists

Re: KafkaDirectStream can't be recovered from checkpoint

Hi Michal, yes, it is there logged twice, it can be seen in attached log in one of previous post with more details: 15/09/17 23:06:37 INFO StreamingContext: Invoking stop(stopGracefully=false) from shutdown hook 15/09/17 23:06:37 INFO StreamingContext: Invoking stop(stopGracefully=false) from

Re: KafkaDirectStream can't be recovered from checkpoint

Also, giving more log4j messages around the error area would be useful. On Thu, Sep 17, 2015 at 1:02 PM, Cody Koeninger wrote: > Is there a particular reason you're calling checkpoint on the stream in > addition to the streaming context? > > On Thu, Sep 17, 2015 at 2:36 PM,

Re: KafkaDirectStream can't be recovered from checkpoint

Hi Petr, after Ctrl+C can you see the following message in the logs? Invoking stop(stopGracefully=false) Details: https://github.com/apache/spark/pull/6307 On 18 September 2015 at 10:28, Petr Novak wrote: > It might be connected with my problems with gracefulShutdown in

Re: KafkaDirectStream can't be recovered from checkpoint

It might be connected with my problems with gracefulShutdown in Spark 1.5.0 2.11 https://mail.google.com/mail/#search/petr/14fb6bd5166f9395 Maybe Ctrl+C corrupts checkpoints and breaks gracefulShutdown? Petr On Fri, Sep 18, 2015 at 4:10 PM, Petr Novak wrote: > ...to

Re: KafkaDirectStream can't be recovered from checkpoint

This one is generated, I suppose, after Ctrl+C 15/09/18 14:38:25 INFO Worker: Asked to kill executor app-20150918143823-0001/0 15/09/18 14:38:25 INFO Worker: Asked to kill executor app-20150918143823-0001/0 15/09/18 14:38:25 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor]

Re: KafkaDirectStream can't be recovered from checkpoint

I have tried it on Spark 1.3.0 2.10 and it works. The same code doesn't on Spark 1.5.0 2.11. It would be nice if anybody could try on another installation to ensure it is something wrong on my cluster. Many thanks, Petr On Fri, Sep 18, 2015 at 4:07 PM, Petr Novak wrote: >

Re: KafkaDirectStream can't be recovered from checkpoint

...to ensure it is not something wrong on my cluster. On Fri, Sep 18, 2015 at 4:09 PM, Petr Novak wrote: > I have tried it on Spark 1.3.0 2.10 and it works. The same code doesn't on > Spark 1.5.0 2.11. It would be nice if anybody could try on another > installation to

KafkaDirectStream can't be recovered from checkpoint

Hi all, it throws FileBasedWriteAheadLogReader: Error reading next item, EOF reached java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.spark.streaming.util.FileBaseWriteAheadLogReader.hasNext(FileBasedWriteAheadLogReader.scala:47) WAL is not

Re: KafkaDirectStream can't be recovered from checkpoint

Is there a particular reason you're calling checkpoint on the stream in addition to the streaming context? On Thu, Sep 17, 2015 at 2:36 PM, Petr Novak wrote: > Hi all, > it throws FileBasedWriteAheadLogReader: Error reading next item, EOF > reached > java.io.EOFException >