Hi Ross, please have a look at Kafka-346, too. In combination with Kafka-345 our scenario, which should be a lot like yours, is covered. Both patches are applied to the github-branch hmb mentioned.
Greetings Peter Am 22.05.2012 17:45 schrieb "Hisham Mardam-Bey" <his...@mate1inc.com>: > Hi Ross, > > A similar thread[1] was just discussed on the list here and resulted in: > > https://issues.apache.org/jira/browse/KAFKA-345 > > and > > > https://github.com/optivo-org/kafka/commit/c4b2647101ab857dda4cb831863dd37e5cb4df55 > > [1] > http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201205.mbox/browser > > Hope this sheds some light on your question, > > Best, > > hmb. > > On Tue, May 22, 2012 at 9:00 AM, Ross Black <ross.w.bl...@gmail.com> > wrote: > > Hi, > > > > I am evaluating kafka to use in our application, and had some questions > > about allocation of partitions to consumers. > > We want to partition messages across a set of consumers so that ideally > > each consumer handles a fixed set of ids (contained with the messages). > > Each of our consumers maintains state for the set of ids it processes. > > > > As I understand it, using a custom Partitioner will allow allocation of > > messages to partitions, and then each consumer will be allocated one or > > more partitions to process. After a change to the number of brokers or > > consumers the allocation of partitions to consumers will change so that > > each consumer may now end up processing a different subset of messages. > > > > Is there some facility within kafka that would allow the set of ids to > > remain fixed for a particular consumer? (From what I have read I assume > > that this is not possible). > > > > Alternatively is there any callback or other notification mechanism that > > would allow our consumers to know when the partitioning changes? > > Since each of our consumers maintains state for the set of ids it is > > processing, we could then dump and refresh that state when the set of ids > > change. > > > > Thanks, > > Ross > > > > -- > Hisham Mardam-Bey > [ Director of Engineering ] [ Mate1 Inc. ] > > A: Because it messes up the order in which people normally read text. > Q: Why is top-posting such a bad thing? > A: Top-posting. > Q: What is the most annoying thing in e-mail? > > -=[ Codito Ergo Sum ]=- >
