Hi Ross, A similar thread[1] was just discussed on the list here and resulted in:
https://issues.apache.org/jira/browse/KAFKA-345 and https://github.com/optivo-org/kafka/commit/c4b2647101ab857dda4cb831863dd37e5cb4df55 [1] http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201205.mbox/browser Hope this sheds some light on your question, Best, hmb. On Tue, May 22, 2012 at 9:00 AM, Ross Black <ross.w.bl...@gmail.com> wrote: > Hi, > > I am evaluating kafka to use in our application, and had some questions > about allocation of partitions to consumers. > We want to partition messages across a set of consumers so that ideally > each consumer handles a fixed set of ids (contained with the messages). > Each of our consumers maintains state for the set of ids it processes. > > As I understand it, using a custom Partitioner will allow allocation of > messages to partitions, and then each consumer will be allocated one or > more partitions to process. After a change to the number of brokers or > consumers the allocation of partitions to consumers will change so that > each consumer may now end up processing a different subset of messages. > > Is there some facility within kafka that would allow the set of ids to > remain fixed for a particular consumer? (From what I have read I assume > that this is not possible). > > Alternatively is there any callback or other notification mechanism that > would allow our consumers to know when the partitioning changes? > Since each of our consumers maintains state for the set of ids it is > processing, we could then dump and refresh that state when the set of ids > change. > > Thanks, > Ross -- Hisham Mardam-Bey [ Director of Engineering ] [ Mate1 Inc. ] A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing in e-mail? -=[ Codito Ergo Sum ]=-
