That's an interesting use case and it is not what the ZK consumer is really designed for. First, having a topic per user may not be ideal, especially if you have millions of users. Second, the ZK consumer is designed for continuously consuming data, not skipping data all the time.
With that said, you can probably try the following trick and see if it works. Set both the log segment size and the log retention size to a value that matches the size of about 100 messages. This should make sure that at any given point time, only the most recent 100 messages are kept in the broker. Set autooffset.reset to earliest. This will make sure that if the consumer's offset is out of range, it will automatically switch to the current smallest offset. Thanks, Jun On Mon, Sep 10, 2012 at 9:07 PM, Ajit Koti <ajitk...@gmail.com> wrote: > Hi, > > No Consumers are not falling behind . > What happens is the users of the consumers don't fetch messages for > certain period of time and when then the users want to fetch the want to > fetch the latest messages from the topic. > > For example in case of displaying news feed each user has got its own topic > , if the users don't login for couple of day > he wouldn't consume the messages from the topic.But when he logs in he want > to get the latest update for which we have to fetch the latest messages > from the topic. > > So is it possible to use the zkconsumer and set the offset to last 100 > messages and then fetch them. > > Thanks > Ajit Koti > > > On Tue, Sep 11, 2012 at 9:04 AM, Jun Rao <jun...@gmail.com> wrote: > > > Is consuming falling behind? Can you have more consumers in the same > group > > to increase the consumption throughput? > > > > Thanks, > > > > Jun > > > > On Mon, Sep 10, 2012 at 12:22 PM, Ajit Koti <ajitk...@gmail.com> wrote: > > > > > Hi All , > > > > > > Was happy consuming messages from the consumer . > > > Recently got a new requirement where I have to process the latest > > message > > > always . > > > Is there anyways I can fetch the latest 100 messages.I know apart from > > > simple consumer I cannot specify the offset . > > > But still wondering is there any way to start consuming from a > particular > > > offset. > > > > > > > > > Note : I have only one consumer per topic. > > > > > > > > > > > > Thanks > > > Ajit > > > > > >
