[
https://issues.apache.org/jira/browse/KAFKA-7358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
M. Manna updated KAFKA-7358:
----------------------------
Description:
In my organisation, we have been using kafka as the basic publish-subscribe
messaging system provider. Our goal is the send event-based (secure, encrypted)
SQL messages reliably, and process them accordingly. For us, the message keys
represent some metadata which we use to either ignore messages (if a loopback
to the sender), or log some information. We have the following use case for
messaging:
1) A Database transaction event takes place
2) The event is captured and messaged across 10 data centres all around the
world.
3) A group of consumers (for each data centre with a unique consumer-group ID)
are will process messages from their respective partitions. 1 consumer per
partition.
Under the circumstances, we only need a guarantee that same message won't be
sent to multiple partitions. In other words, 1 partition will +never+ be sought
by multiple consumers.
Using DefaultPartitioner, we can achieve this only with NULL keys. But since we
need keys for metadata, we cannot maintain "Round-robin" selection of
partitions because a key hash will determine which partition to choose. We need
to have round-robin style selection regardless of key type (NULL or not-NULL)
was:
In my organisation, we have been using kafka as the basic publish-subscribe
messaging system provider. Our goal is the send event-based messages reliably
and securely, and perform data synchronisation based on the messages. For us,
the message keys represent some metadata which we use to either ignore messages
(if a loopback) or log some information. We have the following use case for
messaging:
1) A Database transaction event takes place
2) The event is captured and messaged across 10 data centres all around the
world.
3) A group of consumers (for each data centre with a unique consumer-group ID)
are will process messages from their respective partitions. 1 consumer per
partition.
Under the circumstances, we only need a guarantee that same message won't be
sent to multiple partitions. Using DefaultPartitioner, we can achieve this only
with NULL keys. But since we need keys for metadata, we cannot maintain
"Round-robin" selection of partitions because a key hash will determine which
partition to choose. We need both non-null key with round-robin partition
selection for KafkaProducer.
We believe this solution is achievable and stable, because multiple consumer
will not be consuming from the same partition as per Kafka's fundamental
architecture. Hence, the ask for an extended partitioner.
> Extended Partitioner to Support "Always Round Robin" Selection
> --------------------------------------------------------------
>
> Key: KAFKA-7358
> URL: https://issues.apache.org/jira/browse/KAFKA-7358
> Project: Kafka
> Issue Type: Wish
> Components: clients
> Reporter: M. Manna
> Assignee: M. Manna
> Priority: Minor
>
> In my organisation, we have been using kafka as the basic publish-subscribe
> messaging system provider. Our goal is the send event-based (secure,
> encrypted) SQL messages reliably, and process them accordingly. For us, the
> message keys represent some metadata which we use to either ignore messages
> (if a loopback to the sender), or log some information. We have the following
> use case for messaging:
> 1) A Database transaction event takes place
> 2) The event is captured and messaged across 10 data centres all around the
> world.
> 3) A group of consumers (for each data centre with a unique consumer-group
> ID) are will process messages from their respective partitions. 1 consumer
> per partition.
> Under the circumstances, we only need a guarantee that same message won't be
> sent to multiple partitions. In other words, 1 partition will +never+ be
> sought by multiple consumers.
> Using DefaultPartitioner, we can achieve this only with NULL keys. But since
> we need keys for metadata, we cannot maintain "Round-robin" selection of
> partitions because a key hash will determine which partition to choose. We
> need to have round-robin style selection regardless of key type (NULL or
> not-NULL)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
