Re: kafka partition_assignment_strategy

2021-07-01 Thread Ran Lupovich
that's sorts things out בתאריך יום ה׳, 1 ביולי 2021, 18:24, מאת Marcus Schäfer ‏< marcus.schae...@gmail.com>: > Hi, > > > Hi Marcus, > > What Ran meant is that please run the following command to check the > > result. > > > > bin/kafka-topics.sh --describe --topic TOPIC_NAME --bootstrap-server

Re: kafka partition_assignment_strategy

2021-07-01 Thread Marcus Schäfer
Hi, > Hi Marcus, > What Ran meant is that please run the following command to check the > result. > > bin/kafka-topics.sh --describe --topic TOPIC_NAME --bootstrap-server > localhost:9092 Ah, from the kafka tooling. Sorry I was confused because the aws api name their commands also

Re: kafka partition_assignment_strategy

2021-07-01 Thread Luke Chen
Hi Marcus, What Ran meant is that please run the following command to check the result. bin/kafka-topics.sh --describe --topic TOPIC_NAME --bootstrap-server localhost:9092 Thanks. Luke On Thu, Jul 1, 2021 at 8:06 PM Ran Lupovich wrote: > I meant kafka-topics describe command , I want to

Re: kafka partition_assignment_strategy

2021-07-01 Thread Ran Lupovich
I meant kafka-topics describe command , I want to make sure your topic really have 20 partitions בתאריך יום ה׳, 1 ביולי 2021, 14:12, מאת Marcus Schäfer ‏< marcus.schae...@gmail.com>: > Hi, > > > How many partitions do you have for the topic? Can you issue --describe > > please and share, thanks

Re: kafka partition_assignment_strategy

2021-07-01 Thread Marcus Schäfer
Hi, > How many partitions do you have for the topic? Can you issue --describe > please and share, thanks I've assigned 20 partitions: aws kafka describe-configuration-revision --region eu-central-1 --arn "..." --revision 1 base64 --decode ... auto.create.topics.enable=false

Re: kafka partition_assignment_strategy

2021-07-01 Thread Ran Lupovich
How many partitions do you have for the topic? Can you issue --describe please and share, thanks בתאריך יום ה׳, 1 ביולי 2021, 13:31, מאת Marcus Schäfer ‏< marcus.schae...@gmail.com>: > Hi, > > > Your first understanding is correct, provided each “consumer” means a > > “consumer thread” > > all

Re: kafka partition_assignment_strategy

2021-07-01 Thread Marcus Schäfer
Hi, > Your first understanding is correct, provided each “consumer” means a > “consumer thread” all right, thanks > IMO, Second understanding about message distribution is incorrect because > there is something called as max poll records for each consumer. Its 500 by > default. And the time

Re: kafka partition_assignment_strategy

2021-06-30 Thread Shilin Wu
On which messages goes to which partition, it is decided by producer, so using round robin assignor doesn't help here in theory. And for producer, according to https://kafka.apache.org/documentation/ - The DefaultPartitioner now uses a sticky partitioning strategy. This means that records

Re: kafka partition_assignment_strategy

2021-06-30 Thread sunil chaudhari
Hi Marcus, Your first understanding is correct, provided each “consumer” means a “consumer thread” IMO, Second understanding about message distribution is incorrect because there is something called as max poll records for each consumer. Its 500 by default. And the time between 2 polls is also

kafka partition_assignment_strategy

2021-06-30 Thread Marcus Schäfer
Hi, I was reading through the docs and would like to make use of kafka's dynamic partition assignment feature in a way that partitions gets distributed evenly across all consumers. To my understanding the RoundRobinAssignor would be sufficient and I left the problem of rebalance if a consumer