Hi Jun, >From the paper it appears that what I'm looking for is scheduled as "future" >work. I think this paragraph illustrates the functionality we're looking for:
"At the lowest level this is supported by semantically partitioning messages on the join key during publishing so that all messages sent with a particular key go to the same partition and hence arrive at a single consumer process. This provides the foundation for processing distributed streams across a cluster of consumer machines." We need to guarantee that all messages in a single partition are delivered to the _same_ consumer process. Am I correct in understanding that this is not currently possible? One way of achieving this might be to use multiple topics and publish messages to a particular topic which is being consumed by only one consumer process. Am I missing something fundamental or might this work? Thanks for your help, Matt -----Original Message----- From: Jun Rao [mailto:jun...@gmail.com] Sent: Sunday, August 07, 2011 5:30 PM To: kafka-users@incubator.apache.org Subject: Re: More Information On Partitioning You can take a look at the Kafka paper in NetDB ( http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf). If you have more questions after reading, feel free to ask again. Jun On Fri, Aug 5, 2011 at 6:08 PM, Bateman, Matt <mabate...@ebay.com> wrote: > Hi All, > > The documentation on partitioning is a little light on the website. Is > there somewhere I can get more detail as to how this exactly works? > I'm interested from the deployment, load balancing and intended use > patterns perspectives. > > I'm happy to read whatever docs I've missed if pointed in the right > direction. > > Thanks, > > Matt >
