Well, resetting could follow the partition ownership where only owned partitions can be reset by any one consumer. There's definitely a case for consumer applications that are more about observing frequencies rather than total aggregation and so are typically interested in last couple of seconds / minutes of the stream.
On 14 November 2012 05:38, Jun Rao <jun...@gmail.com> wrote: > It's more than that. For example, what if 2 consumers want to set different > offsets for the same partition? > > Jun > > On Tue, Nov 13, 2012 at 4:08 PM, Michal Haris <michal.ha...@visualdna.com > >wrote: > > > Of course assuming that none of the consumers in the group are interested > > in the missed messages in such case. > > On Nov 12, 2012 6:03 AM, "Jun Rao" <jun...@gmail.com> wrote: > > > > > It's easy if there is only a single consumer in the group. In general, > > > however, there could be multiple consumers in the same group. So, some > > kind > > > of coordination is needed. > > > > > > Thanks, > > > > > > Jun > > > > > > On Sat, Nov 10, 2012 at 6:41 AM, Michal Haris < > > michal.ha...@visualdna.com > > > >wrote: > > > > > > > Found it in archives, autooffset.reset is only used when the consumer > > > > registering first time or the last committed offset is out-of-range, > so > > > > that means that we have to create direct connection to the zookeeper > > and > > > > set/delete the /consumer/xyz/offset/topic123/*-* before starting the > > > > consumer to get it reset to the head of the stream.. shouldn't be a > big > > > > problem to patch the consumer api with a method to do that right ? > > > > > > > > On 10 November 2012 11:40, Michal Haris <michal.ha...@visualdna.com> > > > > wrote: > > > > > > > > > Really? So what does the autooffset.reset=largest do ? > > > > > On Nov 9, 2012 3:30 PM, "Jun Rao" <jun...@gmail.com> wrote: > > > > > > > > > >> Bob, > > > > >> > > > > >> The high level consumer doesn't support resetting the starting > > offset > > > > >> right > > > > >> now. We will look into that post 0.8 as part of the consumer > > redesign. > > > > >> > > > > >> Thanks, > > > > >> > > > > >> Jun > > > > >> > > > > >> > > > > >> On Fri, Nov 9, 2012 at 5:04 AM, Bob Cotton <bcot...@rallydev.com> > > > > wrote: > > > > >> > > > > >> > We have a topic whose consumers, on startup, only need to > consume > > > from > > > > >> the > > > > >> > head of the topic. They don't need to consume old messages they > > may > > > > have > > > > >> > missed. > > > > >> > > > > > >> > We would like to use the high-level consumer, as the API is a > bit > > > > >> simpler. > > > > >> > > > > > >> > Is there a way to reset the current offsets for the high-level > > > > consumer? > > > > >> > > > > > >> > We've though about deleting the consumer group nodes in ZK > > manually, > > > > but > > > > >> > that seems brittle as those locations may change from release to > > > > >> release. > > > > >> > > > > > >> > Is there this level of control somewhere I've missed? > > > > >> > > > > > >> > Thanks > > > > >> > - Bob > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > -- > > > > Michal Haris > > > > Software Engineer > > > > > > > > www.visualdna.com | t: +44 (0) 207 734 7033 > > > > > > > > > > -- Michal Haris Software Engineer www.visualdna.com | t: +44 (0) 207 734 7033
