I've created a topic with 4 partitions, and monitor-brokers reports 4
topics:
===================================================================================================================
||COUNT |TOPIC |BUNDLE |PRODUCER |CONSUMER
|BUNDLE + |BUNDLE - ||
|| |4 |1 |8 |0
|0 |0 ||
||RAW SYSTEM |CPU % |MEMORY % |DIRECT % |BW IN %
|BW OUT % |MAX % ||
|| |2.95 |18.36 |1.56 |0.16
|0.29 |18.36 ||
||ALLOC SYSTEM |CPU % |MEMORY % |DIRECT % |BW IN %
|BW OUT % |MAX % ||
|| |42.68 |3.88 | |3.57
|2.90 |42.68 ||
||RAW MSG |MSG/S IN |MSG/S OUT |TOTAL |KB/S IN
|KB/S OUT |TOTAL ||
|| |1500.41 |0.00 |1500.41 |16.14
|29.18 |45.32 ||
||ALLOC MSG |MSG/S IN |MSG/S OUT |TOTAL |KB/S IN
|KB/S OUT |TOTAL ||
|| |3295.35 |118.70 |3414.05 |357.11
|289.76 |646.86 ||
===================================================================================================================
I also see a throughput of over 1k on one of the brokers:
2017-10-10 21:16:25,548 - INFO - [main:BrokerMonitor@203] - Overall Broker
Data:
***************************************************************************************************************************************
||BROKER |BUNDLE
|MSG/S |LONG/S |KB/S |MAX % ||
||ip-[redacted].us-west-2.compute.internal:8080 |0
|0.00 |0.00 |0.00 |5.81 ||
||ip-[redacted].us-west-2.compute.internal:8080 |1
|1500.41 |639.99 |3414.49 |15.97 ||
||TOTAL |1
|1500.41 |3414.49 |639.99 |15.97 ||
***************************************************************************************************************************************
On Tue, Oct 10, 2017 at 3:48 PM, Rajan Dhabalia <[email protected]>
wrote:
> Hi Ryan,
>
> >> I've set "loadBalancerAutoBundleSplitEnabled" to "true" and
> "loadBalancerNamespaceBundleMaxMsgRate" to 1000. I then ran 2 producers
> at 1k msg/s for ~5mins, but I didn't see a bundle split
>
> LoadBalancer will split the bundle only if it has more than 1 topic in the
> bundle (because bundle is a logical part of namespace that contains topics.
> if namespace has only 1 topic then there is no need of split bundle).
> Load-balancer splits bundle if bundle reaches one of the threshold
> configured at broker-config
> <https://git.corp.yahoo.com/cloud-messaging/pulsar/blob/yahoo/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java#L260-L266>:
>
> 1. *loadBalancerNamespaceBundleMaxTopics*: maximum
> topics in a bundle
> 2. *loadBalancerNamespaceBundleMaxSessions*: maximum
> sessions (producers + consumers) in a bundle
> 3. *loadBalancerNamespaceBundleMaxMsgRate*: maximum
> msgRate (in + out) in a bundle
> 4. *loadBalancerNamespaceBundleMaxBandwidthMbytes*: maximum bandwidth
> (in + out) in a bundle
>
> >> I found "bin/pulsar-perf monitor-brokers"
> Using this utility can you confirm bundle usage and can you confirm if it
> meets that threshold to split the bundle?
>
> Thanks,
> Rajan
>
>
>
> On Tue, Oct 10, 2017 at 3:33 PM, Ryan Stout <[email protected]> wrote:
>
>> Hey Pulsar devs,
>>
>> I've deployed a small Pulsar cluster (in AWS) with 2 brokers and 3
>> bookies. I've started doing perf testing using bin/pulsar-perf to determine
>> the limitations of Pulsar. I'm at the point where I can't produce more than
>> ~25k msg/s on a topic (regardless of number of partitions, clients, or
>> bookies). Upon trying to understand the bottleneck, I found
>> "bin/pulsar-perf monitor-brokers" and it showed that only one of the two
>> brokers is receiving traffic. I've set-up the service-discovery service
>> that came with Pulsar, which my producers are hitting, so I expected the
>> requests to be distributed fairly across the brokers, but this is not the
>> case.
>>
>> In conf/broker.conf, there's a load balancing section that seems to hint
>> at the ability for brokers to shed traffic to other brokers. I've tried
>> tuning the values in this section, but haven't been able to get the brokers
>> to share the load. For example, I've set "loadBalancerAutoBundleSplitEnabled"
>> to "true" and "loadBalancerNamespaceBundleMaxMsgRate" to 1000. I then
>> ran 2 producers at 1k msg/s for ~5mins, but I didn't see a bundle split (I
>> also reduced some of the intervals e.g. "loadBalancerSheddingIntervalMinutes"
>> to 1 minute).
>>
>> Is there a way to configure my Pulsar cluster to balance between my 2
>> brokers? Is there perhaps another, better way I might increase throughput?
>>
>
>
