Re: Spark streaming Kafka receiver WriteAheadLog question

2016-05-02 Thread Renyi Xiong
/2016 03:34:51 am---Hi, Thanks a lot, Cody and Mario, for your > comments. > > From: Renyi Xiong > To: Mario Ds Briggs/India/IBM@IBMIN > Date: 01/05/2016 03:34 am > Subject: Re: Spark streaming Kafka receiver WriteAheadLog question > -- > > > >

Re: Spark streaming Kafka receiver WriteAheadLog question

2016-04-26 Thread Mario Ds Briggs
cc: dev Subject: Re: Spark streaming Kafka receiver WriteAheadLog question If you want to refer back

Re: Spark streaming Kafka receiver WriteAheadLog question

2016-04-25 Thread Cody Koeninger
If you want to refer back to Kafka based on offset ranges, why not use createDirectStream? On Fri, Apr 22, 2016 at 11:49 PM, Renyi Xiong wrote: > Hi, > > Is it possible for Kafka receiver generated WriteAheadLogBackedBlockRDD to > hold corresponded Kafka offset range so that during recovery the R

Spark streaming Kafka receiver WriteAheadLog question

2016-04-22 Thread Renyi Xiong
Hi, Is it possible for Kafka receiver generated WriteAheadLogBackedBlockRDD to hold corresponded Kafka offset range so that during recovery the RDD can refer back to Kafka queue instead of paying the cost of write ahead log? I guess there must be a reason here. Could anyone please help me underst