thetumbled commented on code in PR #21011: URL: https://github.com/apache/pulsar/pull/21011#discussion_r1299869470
########## pip/pip-294.md: ########## @@ -0,0 +1,56 @@ +# Background knowledge + +Load balance module in Pulsar broker rely on zk to store and synchronize metadata about load. Every broker will upload its `LocalBrokerData` to zk, and leader broker will retrieve all `LocalBrokerData` from zk ,generate all `BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk. + + +# Motivation + +As every bundle in the cluster corresponds to a zk node, it is common that there are thousands of zk nodes in a cluster, which results into thousands of read/update operations to zk. This will cause a lot of pressure on zk. + +**As All Load Shedding Algorithm pick bundles from top to bottom based on throughput/msgRate, bundles with low throughput/msgRate are rarely be selected for shedding. So there is no need to update these bundleData to zk frequently.** Review Comment: After the new configuration is introduced, we will stop upload the report for the bundles with under X messages/s or X MB/s until its load increased up to the threashold. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
