Taylor,
You can put anything you want in the key. Perhaps you can pass in the topic
name as part of the key?
Thanks,
Jun
On Tue, Oct 4, 2011 at 8:19 AM, Taylor Gautier <tgaut...@tagged.com> wrote:
> A possibly related question - is it possible to use the interface to
> select a partition based on topic name?
>
>
>
> On Oct 4, 2011, at 7:37 AM, Mathias Herberts <mathias.herbe...@gmail.com>
> wrote:
>
> > Hi,
> >
> > the current Partitioner interface is the following:
> >
> > interface Partitioner<T> {
> > int partition(T key, int numPartitions);
> > }
> >
> > this allows for partition selection based solely on message (or
> > message set) key and number of partitions.
> >
> > Message order guarantees provided by Kafka are per partition, a
> > partition being a pair (brokerid, partitionid).
> >
> > With the current Partitioner interface, the partition selection is a
> > blind process. Some use cases might better benefit from a partition
> > selection process which exposes the brokerid/partitionid info.
> >
> > Imagine a Kafka cluster that currently has 2 brokers, each with one
> > partition. Messages are dispatched to both partitions with a custom
> > Partitioner, sending messages witht the same key to the same
> > partition.
> >
> > Now suppose we add a broker with one partition. With the current
> > Partitioner interface, you cannot ensure that messages with a given
> > key that were sent to a given partition will still be sent to the same
> > partition because you have no info on the brokerid/partitionid that a
> > given partition index will select. In some use cases this can be bad
> > as adding a broker will induce the same kind of disturbance as a
> > broker failure.
> >
> > Could the Partitioner interface be modified to include the sorted list
> > of broker/partition as an additional parameter?
>
