Not using check pointing now. Source is producing 1.2million messages to topic. We are using zookeeper offsets for other downstreams too. That's the reason going with createstream which stores offsets in zookeeper.
Sent from Samsung Mobile. <div>-------- Original message --------</div><div>From: Cody Koeninger <c...@koeninger.org> </div><div>Date:12/08/2016 23:42 (GMT+05:30) </div><div>To: Diwakar Dhanuskodi <diwakar.dhanusk...@gmail.com>, user@spark.apache.org </div><div>Cc: </div><div>Subject: Re: KafkaUtils.createStream not picking smallest offset </div><div> </div>Are you checkpointing? Beyond that, why are you using createStream instead of createDirectStream On Fri, Aug 12, 2016 at 12:32 PM, Diwakar Dhanuskodi <diwakar.dhanusk...@gmail.com> wrote: > Okay . > I could delete the consumer group in zookeeper and start again to re > use same consumer group name. But this is not working though . Somehow > createstream is picking the offset from some where other than > /consumers/ from zookeeper > > > Sent from Samsung Mobile. > > > > > > > > > -------- Original message -------- > From: Cody Koeninger <c...@koeninger.org> > Date:12/08/2016 18:02 (GMT+05:30) > To: Diwakar Dhanuskodi <diwakar.dhanusk...@gmail.com> > Cc: > Subject: Re: KafkaUtils.createStream not picking smallest offset > > Auto offset reset only applies if there aren't offsets available otherwise. > > The old high level consumer stores offsets in zookeeper. > > If you want to make sure you're starting clean, use a new consumer group > I'd. > > On Aug 12, 2016 3:35 AM, "Diwakar Dhanuskodi" <diwakar.dhanusk...@gmail.com> > wrote: >> >> >> Hi, >> We are using spark 1.6.1 and kafka 0.9. >> >> KafkaUtils.createStream is showing strange behaviour. Though >> auto.offset.reset is set to smallest . Whenever we need to restart >> the stream it is picking up the latest offset which is not expected. >> Do we need to set any other properties ?. >> >> createDirectStream works fine in this above case. >> >> >> Sent from Samsung Mobile.
