M. Manna created KAFKA-7358:
-------------------------------
Summary: Extended Partitioner to Support "Always Round Robin"
Selection
Key: KAFKA-7358
URL: https://issues.apache.org/jira/browse/KAFKA-7358
Project: Kafka
Issue Type: Wish
Components: clients
Reporter: M. Manna
Assignee: M. Manna
In my organisation, we have been using kafka as the basic publish-subscribe
messaging system provider. Our goal is the send event-based messages reliably
and securely, and perform data synchronisation based on the messages. For us,
the message keys represent some metadata which we use to either ignore messages
(if a loopback) or log some information. We have the following use case for
messaging:
1) A Database transaction event takes place
2) The event is captured and messaged across 10 data centres all around the
world.
3) A group of consumers (for each data centre with a unique consumer-group ID)
are will process messages from their respective partitions. 1 consumer per
partition.
Under the circumstances, we only need a guarantee that same message won't be
sent to multiple partitions. Using DefaultPartitioner, we can achieve this only
with NULL keys. But since we need keys for metadata, we cannot maintain
"Round-robin" selection of partitions because a key hash will determine which
partition to choose. We need both non-null key with round-robin partition
selection for KafkaProducer.
We believe this solution is achievable and stable, because multiple consumer
will not be consuming from the same partition as per Kafka's fundamental
architecture. Hence, the ask for an extended partitioner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
