Hey Pulsar devs, I've deployed a small Pulsar cluster (in AWS) with 2 brokers and 3 bookies. I've started doing perf testing using bin/pulsar-perf to determine the limitations of Pulsar. I'm at the point where I can't produce more than ~25k msg/s on a topic (regardless of number of partitions, clients, or bookies). Upon trying to understand the bottleneck, I found "bin/pulsar-perf monitor-brokers" and it showed that only one of the two brokers is receiving traffic. I've set-up the service-discovery service that came with Pulsar, which my producers are hitting, so I expected the requests to be distributed fairly across the brokers, but this is not the case.
In conf/broker.conf, there's a load balancing section that seems to hint at the ability for brokers to shed traffic to other brokers. I've tried tuning the values in this section, but haven't been able to get the brokers to share the load. For example, I've set "loadBalancerAutoBundleSplitEnabled" to "true" and "loadBalancerNamespaceBundleMaxMsgRate" to 1000. I then ran 2 producers at 1k msg/s for ~5mins, but I didn't see a bundle split (I also reduced some of the intervals e.g. "loadBalancerSheddingIntervalMinutes" to 1 minute). Is there a way to configure my Pulsar cluster to balance between my 2 brokers? Is there perhaps another, better way I might increase throughput?
