Hi Neha, What does it mean to fetch the latest data? If there are items in the kafka queue and no new items are coming in, would autooffset.reset="largest" wait for new items or would it consume some of the existing items in the queue?
I'm asking this because I've modified the sensei gateways to autooffset.reset="largest" but it's still consuming items previously in the queue, and I'm not sure if the problem is on my end. I know I can get /consumers/[consumer]/offsets/[topic]/, but can I find the latest offset for the queue itself (not the consumer)? Elben On Thu, Mar 22, 2012 at 9:05 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > John, > > You can achieve that via the autooffset.reset="largest" config option > in the consumer. > > Thanks, > Neha > > On Thu, Mar 22, 2012 at 6:15 PM, John Wang <john.w...@gmail.com> wrote: > > Hi Neha: > > > > Can you point me to the api to always get data from tail? > > > > Thanks > > > > -John > > > > On Thu, Mar 22, 2012 at 5:02 PM, Neha Narkhede <neha.narkh...@gmail.com > >wrote: > > > >> Elben, > >> > >> Or sensei can provide a restart from tail API, that will delete the > offset > >> state from ZK and restart its Kafka consumer. Currently that is the best > >> way to have a Kafka consumer restart fetching latest data. > >> > >> Thanks, > >> Neha > >> On Mar 22, 2012 4:39 PM, "John Wang" <john.w...@gmail.com> wrote: > >> > >> > I think in the newer kafka release, there is a tool to update the > version > >> > for a client. > >> > > >> > Jun, can you elaborate on that? > >> > > >> > Thanks > >> > > >> > -John > >> > > >> > On Thu, Mar 22, 2012 at 4:33 PM, Elben Shira <elbensh...@gmail.com> > >> wrote: > >> > > >> > > We're using use kafka+sensei at our company and we'd love to be > able to > >> > > start at the end of the queue (latest entries). The use case is when > >> > sensei > >> > > goes down or has to be restarted, we only care about indexing the > >> latest > >> > > items. > >> > > > >> > > A workaround I guess is to stop sensei, manually change the high > >> > watermark > >> > > in kafka, then restart sensei. > >> > > > >> > > Elben > >> > > > >> > > > >> > > On Thu, Mar 15, 2012 at 11:31 AM, Jun Rao <jun...@gmail.com> wrote: > >> > > > >> > > > Casey, > >> > > > > >> > > > You can't get offsets using zk-based consumer now. We are still > >> trying > >> > to > >> > > > figure about how to support that in the future. > >> > > > > >> > > > Thanks, > >> > > > > >> > > > Jun > >> > > > > >> > > > On Thu, Mar 15, 2012 at 7:57 AM, Sybrandy, Casey < > >> > > > casey.sybra...@six3systems.com> wrote: > >> > > > > >> > > > > Hello, > >> > > > > > >> > > > > Just out of curiosity, how would you get the offsets when using > a > >> > > > zk-based > >> > > > > consumer? I don't see anything that seems to let you do this > >> easily. > >> > > > > > >> > > > > Casey > >> > > > > > >> > > > > ________________________________________ > >> > > > > From: Joel Koshy [jjkosh...@gmail.com] > >> > > > > Sent: Monday, March 12, 2012 5:51 PM > >> > > > > To: kafka-users@incubator.apache.org > >> > > > > Subject: Re: Replay > >> > > > > > >> > > > > With simple consumer, you can start from any valid offset that > you > >> > keep > >> > > > > track off. > >> > > > > > >> > > > > With the high level (zk-based) consumer: you will need to stop > all > >> > > > > consumers, and then update the offsets in zookeeper (see the > >> > > > > ImportZkOffsets tool in the tools package) with the offsets you > >> want > >> > to > >> > > > > replay from before restarting your consumer. > >> > > > > > >> > > > > Thanks, > >> > > > > > >> > > > > Joel > >> > > > > > >> > > > > On Mon, Mar 12, 2012 at 12:09 PM, Jain, Kokil < > ja...@bit-sys.com> > >> > > wrote: > >> > > > > > >> > > > > > Hi, > >> > > > > > > >> > > > > > Sorry, new to Kafka. Is it possible for a consumer to replay > >> > messages > >> > > > > from > >> > > > > > an "offset" that may have been committed before? > >> > > > > > > >> > > > > > Thanks for the help, > >> > > > > > Kokil Jain > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> >
