[
https://issues.apache.org/jira/browse/KAFKA-14269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mathieu Amblard updated KAFKA-14269:
------------------------------------
Description:
*The context :*
I have only one type of message per topic, multiple consumers in the same
microservice and I am using a strategy where I stopped the consumer if the
consumption failed.
*The need :*
I would like to have a Topic Round Robin Assignor in order to continue to
consume other topics even if one failed.
If there are exactly the same number of consumers and topics, then each
consumer will get all partitions of one topic.
If there are more consumers than topics, then each consumer will get a small
number of partitions of one topic.
If there are less consumers than topics, then each consumer will get all
partitions of one topic, some topics will not be consumed at all.
As far as I know, there are currently 4 different strategies :
CooperativeStickyAssignor, RangeAssignor, RoundRobinAssignor, StickyAssignor.
Therefore, I have written my own Topic Round Robin Assignor that assigns all
partitions from each topic to one specific consumer.
For example, suppose there are two consumers *C0* and {*}C1{*}, two topics *t0*
and {*}t1{*}, and each topic has 3 partitions, resulting in partitions
{*}t0p0{*}, {*}t0p1{*}, {*}t0p2{*}, {*}t1p0{*}, {*}t1p1{*}, and {*}t1p2{*}.
The assignment will be:
C0: [t0p0, t0p1 t0p1]
C1: [t1p0, t1p1, t1p2]
First of all, I would like to know if this is a legitimate need.
If this is the case, if you are interested to have a Pull Request about it.
Thank you in advance.
was:
*The context :*
I have only one type of message per topic, multiple consumers in the same
microservice and I am using a strategy where I stopped the consumer if the
consumption failed.
*The need :*
I would like to have a Topic Round Robin Assignor in order to continue to
consume other topics even if one failed.
As far as I know, there are currently 4 different strategies :
CooperativeStickyAssignor, RangeAssignor, RoundRobinAssignor, StickyAssignor.
Therefore, I have written my own Topic Round Robin Assignor that assigns all
partitions from each topic to one specific consumer.
For example, suppose there are two consumers *C0* and {*}C1{*}, two topics *t0*
and {*}t1{*}, and each topic has 3 partitions, resulting in partitions
{*}t0p0{*}, {*}t0p1{*}, {*}t0p2{*}, {*}t1p0{*}, {*}t1p1{*}, and {*}t1p2{*}.
The assignment will be:
C0: [t0p0, t0p1 t0p1]
C1: [t1p0, t1p1, t1p2]
First of all, I would like to know if this is a legitimate need.
If this is the case, if you are interested to have a Pull Request about it.
Thank you in advance.
> Partition Assignment Strategy - Topic Round Robin Assignor
> ----------------------------------------------------------
>
> Key: KAFKA-14269
> URL: https://issues.apache.org/jira/browse/KAFKA-14269
> Project: Kafka
> Issue Type: Wish
> Reporter: Mathieu Amblard
> Priority: Major
>
> *The context :*
> I have only one type of message per topic, multiple consumers in the same
> microservice and I am using a strategy where I stopped the consumer if the
> consumption failed.
> *The need :*
> I would like to have a Topic Round Robin Assignor in order to continue to
> consume other topics even if one failed.
> If there are exactly the same number of consumers and topics, then each
> consumer will get all partitions of one topic.
> If there are more consumers than topics, then each consumer will get a small
> number of partitions of one topic.
> If there are less consumers than topics, then each consumer will get all
> partitions of one topic, some topics will not be consumed at all.
>
> As far as I know, there are currently 4 different strategies :
> CooperativeStickyAssignor, RangeAssignor, RoundRobinAssignor, StickyAssignor.
> Therefore, I have written my own Topic Round Robin Assignor that assigns all
> partitions from each topic to one specific consumer.
> For example, suppose there are two consumers *C0* and {*}C1{*}, two topics
> *t0* and {*}t1{*}, and each topic has 3 partitions, resulting in partitions
> {*}t0p0{*}, {*}t0p1{*}, {*}t0p2{*}, {*}t1p0{*}, {*}t1p1{*}, and {*}t1p2{*}.
> The assignment will be:
> C0: [t0p0, t0p1 t0p1]
> C1: [t1p0, t1p1, t1p2]
>
> First of all, I would like to know if this is a legitimate need.
> If this is the case, if you are interested to have a Pull Request about it.
> Thank you in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
