lhotari edited a comment on pull request #12137: URL: https://github.com/apache/pulsar/pull/12137#issuecomment-925285825
> user can always enable auto bundle unloading in loadbalancer. and default-bundle =4 is not defined for test cluster with 1 namespace but it considers a normal cluster that is serving multiple namespaces with considerable traffic. again, higher number of bundles comes with cost specially for znodes and default behavior unnecessarily will add more bundles for users who are using default conf file. I have a test cluster on GCP/GKE where the auto bundle unloading never happens. It's using the default settings which contains ``` loadBalancerBrokerOverloadedThresholdPercentage=85 loadBalancerAutoUnloadSplitBundlesEnabled=true loadBalancerNamespaceBundleMaxSessions=1000 ``` The test load is created with this type of pulsar-perf command (using 2.7.3 pulsar-perf since current [2.8.0 contains a bug that is fixed in master](https://github.com/apache/pulsar/pull/11824), the bug causes partitions to stop consuming in pulsar-perf). ``` # create a 100 partition topic pulsar-admin topics create-partitioned-topic -p 100 persistent://public/default/perftest # produce messages PULSAR_EXTRA_OPTS="-Xms1500M -Xmx1500M -XX:MaxDirectMemorySize=768M" /pulsar/bin/pulsar-perf produce -u pulsar+ssl://pulsar-broker:6651/ -ioThreads 4 -s 512 -r 50000 persistent://public/default/perftest # consume messages PULSAR_EXTRA_OPTS="-Xms1500M -Xmx1500M -XX:MaxDirectMemorySize=768M" /pulsar/bin/pulsar-perf consume -u pulsar+ssl://pulsar-broker:6651/ -s sub persistent://public/default/perftest ``` (these are run on a pod which has 2CPU + 4GB RAM requested/limits, using pulsar-perf from Pulsar 2.7.3 docker image) The traffic can go on for hours and the CPU load will be very low. Since the threshold never gets exceeded, the auto unload and splitting of the bundle never happens. In this case, the GKE cluster has a nodepool with 6 n2-standard-32 VMs (32 cores, 128GB RAM). Storage for bookies in on SSD. The impact of this is that one of the brokers will never get any real load on it. After changing to a namespace with 32 bundles, the load was evenly distributed. This is the reason why I think that `defaultNumberOfNamespaceBundles=4` is not a good default. The bundle splitting has an open bug #5510 which is another reason to use a higher number of default namespace bundles. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
