A possibly related question - is it possible to use the interface to select a partition based on topic name?
On Oct 4, 2011, at 7:37 AM, Mathias Herberts <mathias.herbe...@gmail.com> wrote: > Hi, > > the current Partitioner interface is the following: > > interface Partitioner<T> { > int partition(T key, int numPartitions); > } > > this allows for partition selection based solely on message (or > message set) key and number of partitions. > > Message order guarantees provided by Kafka are per partition, a > partition being a pair (brokerid, partitionid). > > With the current Partitioner interface, the partition selection is a > blind process. Some use cases might better benefit from a partition > selection process which exposes the brokerid/partitionid info. > > Imagine a Kafka cluster that currently has 2 brokers, each with one > partition. Messages are dispatched to both partitions with a custom > Partitioner, sending messages witht the same key to the same > partition. > > Now suppose we add a broker with one partition. With the current > Partitioner interface, you cannot ensure that messages with a given > key that were sent to a given partition will still be sent to the same > partition because you have no info on the brokerid/partitionid that a > given partition index will select. In some use cases this can be bad > as adding a broker will induce the same kind of disturbance as a > broker failure. > > Could the Partitioner interface be modified to include the sorted list > of broker/partition as an additional parameter?