thetumbled commented on code in PR #21085:
URL: https://github.com/apache/pulsar/pull/21085#discussion_r1314519436
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/ModularLoadManagerImpl.java:
##########
@@ -1139,6 +1143,29 @@ public void writeBrokerDataOnZooKeeper(boolean force) {
}
}
+ /**
+ * sort bundles by load and select topK bundles for each broker.
+ * @return the number of bundles selected
+ */
+ private CompletableFuture<Integer> selectTopKBundle() {
Review Comment:
The original logic is that each follower only uploads its own broker data,
the leader reads and extracts BundleData, aggregates and uploads BundleData.
If the follower broker only uploads topK from the source, it can make the
brokerData smaller. But I feel that such optimization may not be very
significant? The pressure should mainly be caused by the reading and writing of
a large number of Znodes, with a large number of Bundles corresponding to a
large number of Znodes.
Moreover, I feel that the current approach also has a benefit, which is to
let the leader broker select the **global** topK * brokerCount, without the
need for each follower to select a **local** top K.
PS: If we adopt the idea that follower broker select topK bundleData, we
still need to select topK * brokerCount bundleData in leader broker, because
each follower uploads only topK does not mean that the BundleData in the leader
broker's memory only stores topK * brokerCount.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
