Elben, What I had suggested was the following -
>> Or sensei can provide a restart from tail API, that will delete the offset >> state from ZK and restart its Kafka consumer. Currently that is the best way >> to have a Kafka consumer restart fetching latest data. Note that sensei would need to first get rid of the current state for the consumer group from ZK (delete /consumers/[your-consumer-group]). And then when you restart the Kafka consumer, it will start consuming from the latest data available on the server. Thanks, Neha On Thu, Mar 29, 2012 at 10:10 AM, Jun Rao <jun...@gmail.com> wrote: > Elben, > > autooffset.reset="largest" is only used for the very first time the > consumer is started when there is no offset registered in ZK. Once the > consumer is started, by default it periodically checkpoints offset in ZK > and on subsequent restarts, the consumer starts consumption from the offset > stored in ZK. > > Thanks, > > Jun > > On Thu, Mar 29, 2012 at 9:54 AM, Elben Shira <elbensh...@gmail.com> wrote: > >> Hi Neha, >> >> What does it mean to fetch the latest data? If there are items in the kafka >> queue and no new items are coming in, would autooffset.reset="largest" wait >> for new items or would it consume some of the existing items in the queue? >> >> I'm asking this because I've modified the sensei gateways >> to autooffset.reset="largest" but it's still consuming items previously in >> the queue, and I'm not sure if the problem is on my end. >> >> I know I can get /consumers/[consumer]/offsets/[topic]/, but can I find the >> latest offset for the queue itself (not the consumer)? >> >> Elben >> >> >> On Thu, Mar 22, 2012 at 9:05 PM, Neha Narkhede <neha.narkh...@gmail.com >> >wrote: >> >> > John, >> > >> > You can achieve that via the autooffset.reset="largest" config option >> > in the consumer. >> > >> > Thanks, >> > Neha >> > >> > On Thu, Mar 22, 2012 at 6:15 PM, John Wang <john.w...@gmail.com> wrote: >> > > Hi Neha: >> > > >> > > Can you point me to the api to always get data from tail? >> > > >> > > Thanks >> > > >> > > -John >> > > >> > > On Thu, Mar 22, 2012 at 5:02 PM, Neha Narkhede < >> neha.narkh...@gmail.com >> > >wrote: >> > > >> > >> Elben, >> > >> >> > >> Or sensei can provide a restart from tail API, that will delete the >> > offset >> > >> state from ZK and restart its Kafka consumer. Currently that is the >> best >> > >> way to have a Kafka consumer restart fetching latest data. >> > >> >> > >> Thanks, >> > >> Neha >> > >> On Mar 22, 2012 4:39 PM, "John Wang" <john.w...@gmail.com> wrote: >> > >> >> > >> > I think in the newer kafka release, there is a tool to update the >> > version >> > >> > for a client. >> > >> > >> > >> > Jun, can you elaborate on that? >> > >> > >> > >> > Thanks >> > >> > >> > >> > -John >> > >> > >> > >> > On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <elbensh...@gmail.com> >> > >> wrote: >> > >> > >> > >> > > We're using use kafka+sensei at our company and we'd love to be >> > able to >> > >> > > start at the end of the queue (latest entries). The use case is >> when >> > >> > sensei >> > >> > > goes down or has to be restarted, we only care about indexing the >> > >> latest >> > >> > > items. >> > >> > > >> > >> > > A workaround I guess is to stop sensei, manually change the high >> > >> > watermark >> > >> > > in kafka, then restart sensei. >> > >> > > >> > >> > > Elben >> > >> > > >> > >> > > >> > >> > > On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <jun...@gmail.com> >> wrote: >> > >> > > >> > >> > > > Casey, >> > >> > > > >> > >> > > > You can't get offsets using zk-based consumer now. We are still >> > >> trying >> > >> > to >> > >> > > > figure about how to support that in the future. >> > >> > > > >> > >> > > > Thanks, >> > >> > > > >> > >> > > > Jun >> > >> > > > >> > >> > > > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey < >> > >> > > > casey.sybra...@six3systems.com> wrote: >> > >> > > > >> > >> > > > > Hello, >> > >> > > > > >> > >> > > > > Just out of curiosity, how would you get the offsets when >> using >> > a >> > >> > > > zk-based >> > >> > > > > consumer? I don't see anything that seems to let you do this >> > >> easily. >> > >> > > > > >> > >> > > > > Casey >> > >> > > > > >> > >> > > > > ________________________________________ >> > >> > > > > From: Joel Koshy [jjkosh...@gmail.com] >> > >> > > > > Sent: Monday, March 12, 2012 5:51 PM >> > >> > > > > To: kafka-users@incubator.apache.org >> > >> > > > > Subject: Re: Replay >> > >> > > > > >> > >> > > > > With simple consumer, you can start from any valid offset that >> > you >> > >> > keep >> > >> > > > > track off. >> > >> > > > > >> > >> > > > > With the high level (zk-based) consumer: you will need to stop >> > all >> > >> > > > > consumers, and then update the offsets in zookeeper (see the >> > >> > > > > ImportZkOffsets tool in the tools package) with the offsets >> you >> > >> want >> > >> > to >> > >> > > > > replay from before restarting your consumer. >> > >> > > > > >> > >> > > > > Thanks, >> > >> > > > > >> > >> > > > > Joel >> > >> > > > > >> > >> > > > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil < >> > ja...@bit-sys.com> >> > >> > > wrote: >> > >> > > > > >> > >> > > > > > Hi, >> > >> > > > > > >> > >> > > > > > Sorry, new to Kafka. Is it possible for a consumer to replay >> > >> > messages >> > >> > > > > from >> > >> > > > > > an "offset" that may have been committed before? >> > >> > > > > > >> > >> > > > > > Thanks for the help, >> > >> > > > > > Kokil Jain >> > >> > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > >> > >> >> > >>
