Yeah, this is a common need that we don't really expose. It would be great to scope out an API or appropriate plugin mechanism for this.
-Jay On Wed, May 2, 2012 at 1:55 AM, Peter Romianowski <honkb...@googlemail.com> wrote: > Hi, > > thanks for the quick answers. I took a look at the source code (I should > have done it before asking this question) and it was quite obvious, that > partitions might get moved. > > In our scenario we partition events by userid and then apply these to some > kind of state machine, that modifies the actual state of a user. So events > trigger state transitions. In order to avoid the need of loading user's > state upon each event processed, we cache that. But if a user's partition > is moved to another consumer and then back to the previous consumer we have > stale caches and hell breaks loose. I guess the same kind of problem occurs > in other scenarios like counting numbers by user, too. > > We would like to stick to the high level consumer and it's rebalancing. > > All that's needed to fullfill our use case is some kind of listener I could > add to the consumer that gets notified whenever rebalancing happens. I > would not even need to know which partitions got moved away and which got > assigned, since I would simply flush the whole cache. During normal > operation rebalancing should only happen, if infrastructure like brokers, > number of partitions or consumer changes, right? > > I guess I'll take a deeper dive into the source code. Any implementation > hints? > > Thanks again > > Peter > Am 02.05.2012 03:23 schrieb "Jun Rao" <jun...@gmail.com>: > >> Peter, >> >> Currently the partition assignment could change when there is any change in >> brokers and consumers in the same group. The change is typically triggered >> by starting or stopping a broker/consumer. However, it can also happen if >> the broker/consumer somehow expires its ZK session (e.g., long GC). >> >> Thanks, >> >> Jun >> >> On Tue, May 1, 2012 at 3:28 PM, Peter Romianowski >> <honkb...@googlemail.com>wrote: >> >> > Hi, >> > >> > wer are using the high-level Java consumer. We feed the events received >> > from Kafka into some sort of state machine. Since we have Kafka's >> guarantee >> > that each partition is read by the same consumer, we want to keep the >> > states in memory to achieve even higher throughput. So it is vital for us >> > that either a partition is never moved from a running consumer or our >> code >> > gets at least informed about that. >> > >> > Does Kafka guarantee that a partition assigned to a consumer will stay at >> > this consumer for the whole lifetime of the jvm? Even in corner cases >> like >> > loosing connection to zookeeper? >> > >> > Regards, >> > >> > Peter >> > >>
