thetumbled commented on code in PR #21085:
URL: https://github.com/apache/pulsar/pull/21085#discussion_r1314519436
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java:
##########
@@ -1139,6 +1143,29 @@ public void writeBrokerDataOnZooKeeper(boolean force) {
}
}
+ /**
+ * sort bundles by load and select topK bundles for each broker.
+ * @return the number of bundles selected
+ */
+ private CompletableFuture<Integer> selectTopKBundle() {
Review Comment:
The original logic is that each follower only uploads its own broker data,
the leader reads and extracts BundleData, aggregates and uploads BundleData.
If the follower broker only uploads the topK bundleData contained in it's
brokerData, it can make the brokerData smaller. But I feel that such
optimization may not be very significant? The pressure should mainly be caused
by the reading and writing of a large number of Znodes, with a large number of
Bundles corresponding to a large number of Znodes.
Moreover, I feel that the current approach(that is only leader broker filter
bundleData) has a benefit, which is to let the leader broker select the
**global** topK * brokerCount, instead of each follower select a **local** top
K.
PS: If we adopt the idea that follower broker select topK bundleData, we
still need to select topK * brokerCount bundleData in leader broker, because
each follower uploads only topK does not mean that the BundleData in the leader
broker's memory only stores topK * brokerCount.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
