Elben, autooffset.reset="largest" is only used for the very first time the consumer is started when there is no offset registered in ZK. Once the consumer is started, by default it periodically checkpoints offset in ZK and on subsequent restarts, the consumer starts consumption from the offset stored in ZK.
Thanks, Jun On Thu, Mar 29, 2012 at 9:54 AM, Elben Shira <elbensh...@gmail.com> wrote: > Hi Neha, > > What does it mean to fetch the latest data? If there are items in the kafka > queue and no new items are coming in, would autooffset.reset="largest" wait > for new items or would it consume some of the existing items in the queue? > > I'm asking this because I've modified the sensei gateways > to autooffset.reset="largest" but it's still consuming items previously in > the queue, and I'm not sure if the problem is on my end. > > I know I can get /consumers/[consumer]/offsets/[topic]/, but can I find the > latest offset for the queue itself (not the consumer)? > > Elben > > > On Thu, Mar 22, 2012 at 9:05 PM, Neha Narkhede <neha.narkh...@gmail.com > >wrote: > > > John, > > > > You can achieve that via the autooffset.reset="largest" config option > > in the consumer. > > > > Thanks, > > Neha > > > > On Thu, Mar 22, 2012 at 6:15 PM, John Wang <john.w...@gmail.com> wrote: > > > Hi Neha: > > > > > > Can you point me to the api to always get data from tail? > > > > > > Thanks > > > > > > -John > > > > > > On Thu, Mar 22, 2012 at 5:02 PM, Neha Narkhede < > neha.narkh...@gmail.com > > >wrote: > > > > > >> Elben, > > >> > > >> Or sensei can provide a restart from tail API, that will delete the > > offset > > >> state from ZK and restart its Kafka consumer. Currently that is the > best > > >> way to have a Kafka consumer restart fetching latest data. > > >> > > >> Thanks, > > >> Neha > > >> On Mar 22, 2012 4:39 PM, "John Wang" <john.w...@gmail.com> wrote: > > >> > > >> > I think in the newer kafka release, there is a tool to update the > > version > > >> > for a client. > > >> > > > >> > Jun, can you elaborate on that? > > >> > > > >> > Thanks > > >> > > > >> > -John > > >> > > > >> > On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <elbensh...@gmail.com> > > >> wrote: > > >> > > > >> > > We're using use kafka+sensei at our company and we'd love to be > > able to > > >> > > start at the end of the queue (latest entries). The use case is > when > > >> > sensei > > >> > > goes down or has to be restarted, we only care about indexing the > > >> latest > > >> > > items. > > >> > > > > >> > > A workaround I guess is to stop sensei, manually change the high > > >> > watermark > > >> > > in kafka, then restart sensei. > > >> > > > > >> > > Elben > > >> > > > > >> > > > > >> > > On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <jun...@gmail.com> > wrote: > > >> > > > > >> > > > Casey, > > >> > > > > > >> > > > You can't get offsets using zk-based consumer now. We are still > > >> trying > > >> > to > > >> > > > figure about how to support that in the future. > > >> > > > > > >> > > > Thanks, > > >> > > > > > >> > > > Jun > > >> > > > > > >> > > > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey < > > >> > > > casey.sybra...@six3systems.com> wrote: > > >> > > > > > >> > > > > Hello, > > >> > > > > > > >> > > > > Just out of curiosity, how would you get the offsets when > using > > a > > >> > > > zk-based > > >> > > > > consumer? I don't see anything that seems to let you do this > > >> easily. > > >> > > > > > > >> > > > > Casey > > >> > > > > > > >> > > > > ________________________________________ > > >> > > > > From: Joel Koshy [jjkosh...@gmail.com] > > >> > > > > Sent: Monday, March 12, 2012 5:51 PM > > >> > > > > To: kafka-users@incubator.apache.org > > >> > > > > Subject: Re: Replay > > >> > > > > > > >> > > > > With simple consumer, you can start from any valid offset that > > you > > >> > keep > > >> > > > > track off. > > >> > > > > > > >> > > > > With the high level (zk-based) consumer: you will need to stop > > all > > >> > > > > consumers, and then update the offsets in zookeeper (see the > > >> > > > > ImportZkOffsets tool in the tools package) with the offsets > you > > >> want > > >> > to > > >> > > > > replay from before restarting your consumer. > > >> > > > > > > >> > > > > Thanks, > > >> > > > > > > >> > > > > Joel > > >> > > > > > > >> > > > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil < > > ja...@bit-sys.com> > > >> > > wrote: > > >> > > > > > > >> > > > > > Hi, > > >> > > > > > > > >> > > > > > Sorry, new to Kafka. Is it possible for a consumer to replay > > >> > messages > > >> > > > > from > > >> > > > > > an "offset" that may have been committed before? > > >> > > > > > > > >> > > > > > Thanks for the help, > > >> > > > > > Kokil Jain > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >
