Hi Karan, I noticed that you posted this to the Confluent Google Group as well. Let's discuss it over there. This mailing list is only for Apache Kafka.
Thanks, Ismael On Wed, Aug 2, 2017 at 7:11 AM, karan alang <karan.al...@gmail.com> wrote: > Hello, here is the update .. > > when i ran script - kafka-preferred-replica-election, it did the > re-election as required. > > > ./bin/kafka-preferred-replica-election --zookeeper localhost:3181 > > so does that mean that i need to run the script -> > ./bin/confluent-rebalancer > to rebalance the data, > but for the leader election, the script to be run is -> > ./bin/kafka-preferred-replica-election > > The documentation mentions this (link - > http://docs.confluent.io/current/kafka/rebalancer/rebalancer.html) > > The confluent-rebalancer tool balances data so that the number of leaders > > and disk usage are even across brokers and racks on a per topic and > cluster > > level while minimising data movement. > > > seems there is a disconnect here, pls let me know if anyone has inputs. > > > On Tue, Aug 1, 2017 at 4:45 PM, karan alang <karan.al...@gmail.com> wrote: > > > Hi All - > > i'm trying to rebalance Kafka topic (refer link -> > > http://docs.confluent.io/current/kafka/rebalancer/rebalancer.html), and > > somehow the rebalancing is not working. > > > > > > Here is what i'm doing -> > > - i've 4 Kafka brokers & i've made changes to the server.properties file > > to enable Confluent Metrics Reporter. > > (attached are the server.properties of the 4 brokers) > > > > -> Created a topic specifying Replica assignment > > > > ./bin/kafka-topics --create --topic topic-a1 --replica-assignment > > 0:1,0:1,0:1,0:1 --zookeeper nwk2-bdp-kafka-04.gdcs-qa.apple.com:3181 > > > > -> describe topic > > > > ./bin/kafka-topics --describe --topic topic-a1 --zookeeper > >> nwk2-bdp-kafka-04.gdcs-qa.apple.com:3181 > >> Topic:topic-a1 PartitionCount:4 ReplicationFactor:2 Configs: > >> Topic: topic-a1 Partition: 0 Leader: 1 Replicas: 1,0 Isr: 0,1 > >> Topic: topic-a1 Partition: 1 Leader: 1 Replicas: 1,0 Isr: 0,1 > >> Topic: topic-a1 Partition: 2 Leader: 0 Replicas: 0,1 Isr: 0,1 > >> Topic: topic-a1 Partition: 3 Leader: 0 Replicas: 0,1 Isr: 0,1 > >> > > > > > > -> Produce data into topics, using the following command > > > > ./bin/kafka-producer-perf-test --topic topic-a1 --num-records 200000 > >> --record-size 1000 --throughput 10000000 --producer-props > bootstrap.servers= > >> nwk2-bdp-kafka-04.gdcs-qa.apple.com:9092,nwk2-bdp-kafka-04.gdcs- > >> qa.apple.com:9082,nwk2-bdp-kafka-04.gdcs-qa.apple.com:9072, > >> nwk2-bdp-kafka-04.gdcs-qa.apple.com:9062 > > > > > > > > -> Force Creation of offsets topic, by creating a Consumer (NOT SURE WHAT > > THIS IS FOR ???) : > > > > ./bin/kafka-consumer-perf-test --topic topic-a1 --broker-list > >> nwk2-bdp-kafka-04.gdcs-qa.apple.com:9092,nwk2-bdp-kafka- > >> 04.gdcs-qa.apple.com:9082,nwk2-bdp-kafka-04.gdcs-qa.apple.com:9072, > >> nwk2-bdp-kafka-04.gdcs-qa.apple.com:9062 --messages 10 > > > > > > > > -> run the following command to rebalance > > > >> > >> The plan that is presented does not really any rebalancing -> > > > > ./bin/confluent-rebalancer execute --zookeeper nwk2-bdp-kafka-04.gdcs-qa. > >> apple.com:3181 --metrics-bootstrap-server nwk2-bdp-kafka-04.gdcs-qa. > >> apple.com:9092,nwk2-bdp-kafka-04.gdcs-qa.apple.com:9082,nwk2 > >> -bdp-kafka-04.gdcs-qa.apple.com:9072,nwk2-bdp-kafka-04. > >> gdcs-qa.apple.com:9062 --throttle 10000000 --verbose > >> > >> Computing the rebalance plan (this may take a while) ... > >> You are about to move 0 replica(s) for 0 partitions to 0 broker(s) with > >> total size 0 MB. > >> The preferred leader for 2 partition(s) will be changed. > >> In total, the assignment for 2 partitions will be changed. > >> The minimum free volume space is set to 20.0%. > >> The following brokers will have less than 40% of free volume space > during > >> the rebalance: > >> Broker Current Size (MB) Size During Rebalance (MB) Free % During > >> Rebalance Size After Rebalance (MB) Free % After Rebalance > >> 0 4,021.1 4,021.1 14.2 > >> 4,021.1 14.2 > >> 1 1,240.8 1,240.8 14.2 > >> 1,240.8 14.2 > >> 2 620.4 620.4 14.2 > >> 620.4 14.2 > >> 3 0 0 14.2 > >> 0 14.2 > >> Min/max stats for brokers (before -> after): > >> Type Leader Count Replica Count Size > >> (MB) > >> Min 0 (id: 3) -> 0 (id: 3) 0 (id: 3) -> 0 (id: 3) 0 (id: > >> 3) -> 0 (id: 3) > >> Max 125 (id: 0) -> 123 (id: 0) 127 (id: 0) -> 127 (id: 0) 4,021.1 > >> (id: 0) -> 4,021.1 (id: 0) > >> No racks are defined. > >> Broker stats (before -> after): > >> Broker Leader Count Replica Count Size (MB) Free > >> Space (%) > >> 0 125 -> 123 127 -> 127 4,021.1 -> 4,021.1 14.2 -> > >> 14.2 > >> 1 3 -> 5 12 -> 12 1,240.8 -> 1,240.8 14.2 -> > >> 14.2 > >> 2 2 -> 2 3 -> 3 620.4 -> 620.4 14.2 -> > >> 14.2 > >> 3 0 -> 0 0 -> 0 0 -> 0 14.2 -> > >> 14.2 > >> Would you like to continue? (y/n): y > >> The rebalance has been started, run `status` to check progress. > >> Warning: You must run the `status` or `finish` command periodically, > >> until the rebalance completes, to ensure the throttle is removed. You > can > >> also alter the throttle by re-running the execute command passing a new > >> value. > >> > >> > > > > -> I describe the topic, after rebalancing > > > > ./bin/kafka-topics --describe --topic topic-a1 --zookeeper > >> nwk2-bdp-kafka-04.gdcs-qa.apple.com:3181 > >> Topic:topic-a1 PartitionCount:4 ReplicationFactor:2 Configs: > >> Topic: topic-a1 Partition: 0 Leader: 1 Replicas: 1,0 Isr: 0,1 > >> Topic: topic-a1 Partition: 1 Leader: 1 Replicas: 1,0 Isr: 0,1 > >> Topic: topic-a1 Partition: 2 Leader: 0 Replicas: 0,1 Isr: 0,1 > >> Topic: topic-a1 Partition: 3 Leader: 0 Replicas: 0,1 Isr: 0,1 > >> > >> > > > > My expectation was that the topic will have all 4 brokers as leaders for > > the 4 partitions, but that does not seem to be happening. > > > > Any ideas what the issue is ? > > > > >
