Hi, Thanks, that info covers exactly what I was looking for.
Are Kafka-345, Kafka-346 intended to be released as part of 0.8, or perhaps earlier? I appreciate the help. Ross On 23 May 2012 08:24, Peter Romianowski <honkb...@googlemail.com> wrote: > Hi Ross, > > please have a look at Kafka-346, too. In combination with Kafka-345 our > scenario, which should be a lot like yours, is covered. Both patches are > applied to the github-branch hmb mentioned. > > Greetings > > Peter > Am 22.05.2012 17:45 schrieb "Hisham Mardam-Bey" <his...@mate1inc.com>: > > > Hi Ross, > > > > A similar thread[1] was just discussed on the list here and resulted in: > > > > https://issues.apache.org/jira/browse/KAFKA-345 > > > > and > > > > > > > https://github.com/optivo-org/kafka/commit/c4b2647101ab857dda4cb831863dd37e5cb4df55 > > > > [1] > > > http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201205.mbox/browser > > > > Hope this sheds some light on your question, > > > > Best, > > > > hmb. > > > > On Tue, May 22, 2012 at 9:00 AM, Ross Black <ross.w.bl...@gmail.com> > > wrote: > > > Hi, > > > > > > I am evaluating kafka to use in our application, and had some questions > > > about allocation of partitions to consumers. > > > We want to partition messages across a set of consumers so that ideally > > > each consumer handles a fixed set of ids (contained with the messages). > > > Each of our consumers maintains state for the set of ids it processes. > > > > > > As I understand it, using a custom Partitioner will allow allocation of > > > messages to partitions, and then each consumer will be allocated one or > > > more partitions to process. After a change to the number of brokers > or > > > consumers the allocation of partitions to consumers will change so that > > > each consumer may now end up processing a different subset of messages. > > > > > > Is there some facility within kafka that would allow the set of ids to > > > remain fixed for a particular consumer? (From what I have read I > assume > > > that this is not possible). > > > > > > Alternatively is there any callback or other notification mechanism > that > > > would allow our consumers to know when the partitioning changes? > > > Since each of our consumers maintains state for the set of ids it is > > > processing, we could then dump and refresh that state when the set of > ids > > > change. > > > > > > Thanks, > > > Ross > > > > > > > > -- > > Hisham Mardam-Bey > > [ Director of Engineering ] [ Mate1 Inc. ] > > > > A: Because it messes up the order in which people normally read text. > > Q: Why is top-posting such a bad thing? > > A: Top-posting. > > Q: What is the most annoying thing in e-mail? > > > > -=[ Codito Ergo Sum ]=- > > >
