Hi, I am evaluating kafka to use in our application, and had some questions about allocation of partitions to consumers. We want to partition messages across a set of consumers so that ideally each consumer handles a fixed set of ids (contained with the messages). Each of our consumers maintains state for the set of ids it processes.
As I understand it, using a custom Partitioner will allow allocation of messages to partitions, and then each consumer will be allocated one or more partitions to process. After a change to the number of brokers or consumers the allocation of partitions to consumers will change so that each consumer may now end up processing a different subset of messages. Is there some facility within kafka that would allow the set of ids to remain fixed for a particular consumer? (From what I have read I assume that this is not possible). Alternatively is there any callback or other notification mechanism that would allow our consumers to know when the partitioning changes? Since each of our consumers maintains state for the set of ids it is processing, we could then dump and refresh that state when the set of ids change. Thanks, Ross
