Hi Ryan, >> I've set "loadBalancerAutoBundleSplitEnabled" to "true" and "loadBalancerNamespaceBundleMaxMsgRate" to 1000. I then ran 2 producers at 1k msg/s for ~5mins, but I didn't see a bundle split
LoadBalancer will split the bundle only if it has more than 1 topic in the bundle (because bundle is a logical part of namespace that contains topics. if namespace has only 1 topic then there is no need of split bundle). Load-balancer splits bundle if bundle reaches one of the threshold configured at broker-config <https://git.corp.yahoo.com/cloud-messaging/pulsar/blob/yahoo/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java#L260-L266>: 1. *loadBalancerNamespaceBundleMaxTopics*: maximum topics in a bundle 2. *loadBalancerNamespaceBundleMaxSessions*: maximum sessions (producers + consumers) in a bundle 3. *loadBalancerNamespaceBundleMaxMsgRate*: maximum msgRate (in + out) in a bundle 4. *loadBalancerNamespaceBundleMaxBandwidthMbytes*: maximum bandwidth (in + out) in a bundle >> I found "bin/pulsar-perf monitor-brokers" Using this utility can you confirm bundle usage and can you confirm if it meets that threshold to split the bundle? Thanks, Rajan On Tue, Oct 10, 2017 at 3:33 PM, Ryan Stout <rst...@box.com> wrote: > Hey Pulsar devs, > > I've deployed a small Pulsar cluster (in AWS) with 2 brokers and 3 > bookies. I've started doing perf testing using bin/pulsar-perf to determine > the limitations of Pulsar. I'm at the point where I can't produce more than > ~25k msg/s on a topic (regardless of number of partitions, clients, or > bookies). Upon trying to understand the bottleneck, I found > "bin/pulsar-perf monitor-brokers" and it showed that only one of the two > brokers is receiving traffic. I've set-up the service-discovery service > that came with Pulsar, which my producers are hitting, so I expected the > requests to be distributed fairly across the brokers, but this is not the > case. > > In conf/broker.conf, there's a load balancing section that seems to hint > at the ability for brokers to shed traffic to other brokers. I've tried > tuning the values in this section, but haven't been able to get the brokers > to share the load. For example, I've set "loadBalancerAutoBundleSplitEnabled" > to "true" and "loadBalancerNamespaceBundleMaxMsgRate" to 1000. I then ran > 2 producers at 1k msg/s for ~5mins, but I didn't see a bundle split (I also > reduced some of the intervals e.g. "loadBalancerSheddingIntervalMinutes" > to 1 minute). > > Is there a way to configure my Pulsar cluster to balance between my 2 > brokers? Is there perhaps another, better way I might increase throughput? >
