This seems workable to me. But how about just using a single consumer and re-assigning topic-partitions on-the-fly then?
Guozhang On Mon, Nov 14, 2016 at 12:17 AM, Enrico Olivelli <eolive...@gmail.com> wrote: > Hi, > I'm using Java Consumer API (latest version, 0.10.1.0), I store > consumed offsets offline, that is that I do not use kafka builtin > offset storage nor consumer groups. > > I need to read data from different topics, but from one topic at once, > I would like to pool my consumers (using something like Apache Commons > Pool). > > My ideal flow is: > - borrow a consumer from a pool (maybe it gets created) > - assign partitions for topic A > - poll > - return the consumer to the pool > > - borrow a consumer > - assign partitions for topic B > - poll > - return the consumer to the pool > > - an so on.... > > I can guarantee that only one thread uses a consumer at once. Any kind > of error will lead to consumer disposition and removal from the pool. > Is such a way of pooling the consumer safe ? Is the a risk of getting > data from topic A while polling from topic B ? > > > Thank you > > Enrico > -- -- Guozhang
