codelipenghui commented on code in PR #21011:
URL: https://github.com/apache/pulsar/pull/21011#discussion_r1299725296
##########
pip/pip-294.md:
##########
@@ -0,0 +1,56 @@
+# Background knowledge
+
+Load balance module in Pulsar broker rely on zk to store and synchronize
metadata about load. Every broker will upload its `LocalBrokerData` to zk, and
leader broker will retrieve all `LocalBrokerData` from zk ,generate all
`BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk.
Review Comment:
Could you please add some context related to the new load manager without zk
dependency? The change will also improve this part, right?
##########
pip/pip-294.md:
##########
@@ -0,0 +1,56 @@
+# Background knowledge
+
+Load balance module in Pulsar broker rely on zk to store and synchronize
metadata about load. Every broker will upload its `LocalBrokerData` to zk, and
leader broker will retrieve all `LocalBrokerData` from zk ,generate all
`BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk.
+
+
+# Motivation
+
+As every bundle in the cluster corresponds to a zk node, it is common that
there are thousands of zk nodes in a cluster, which results into thousands of
read/update operations to zk. This will cause a lot of pressure on zk.
+
+**As All Load Shedding Algorithm pick bundles from top to bottom based on
throughput/msgRate, bundles with low throughput/msgRate are rarely be selected
for shedding. So there is no need to update these bundleData to zk frequently.**
Review Comment:
After the new configuration is introduced, will the frequency of the load
report of bundles without relatively lower throughput be reduced? or will stop
to upload the report for the bundles with under X messages/s or X MB/s.
##########
pip/pip-294.md:
##########
@@ -0,0 +1,56 @@
+# Background knowledge
+
+Load balance module in Pulsar broker rely on zk to store and synchronize
metadata about load. Every broker will upload its `LocalBrokerData` to zk, and
leader broker will retrieve all `LocalBrokerData` from zk ,generate all
`BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk.
+
+
+# Motivation
+
+As every bundle in the cluster corresponds to a zk node, it is common that
there are thousands of zk nodes in a cluster, which results into thousands of
read/update operations to zk. This will cause a lot of pressure on zk.
+
+**As All Load Shedding Algorithm pick bundles from top to bottom based on
throughput/msgRate, bundles with low throughput/msgRate are rarely be selected
for shedding. So there is no need to update these bundleData to zk frequently.**
+
+
+# Goals
+
+Filter out bundles with low throughput/msgRate, and do not update these
bundles to zk frequently to reduce the pressure on zk.
+
+
+# High Level Design
+
+Filter out bundles with low throughput/msgRate when leader update bundleData
to zk.
+
+
+# Detailed Design
+
+## Design & Implementation Details
+Add throughput/msg-based bundle report control in the `ExtensibleLoadManager`
and `ModularLoadManagerImpl`.
+
+### Configuration
+
+add configuration:
+```
+ @FieldContext(
+ dynamic = true,
+ category = CATEGORY_LOAD_BALANCER,
+ doc = "minimum throughput in of bundle to be considered for
updating data in metadata store"
+ )
+ private int loadBalancerBundleThroughputThresholdInByte = 0;
Review Comment:
```suggestion
private int loadBalancerBundleMinThroughputInBytesInBundleLoadReport = 0;
```
##########
pip/pip-294.md:
##########
@@ -0,0 +1,56 @@
+# Background knowledge
+
+Load balance module in Pulsar broker rely on zk to store and synchronize
metadata about load. Every broker will upload its `LocalBrokerData` to zk, and
leader broker will retrieve all `LocalBrokerData` from zk ,generate all
`BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk.
+
+
+# Motivation
+
+As every bundle in the cluster corresponds to a zk node, it is common that
there are thousands of zk nodes in a cluster, which results into thousands of
read/update operations to zk. This will cause a lot of pressure on zk.
+
+**As All Load Shedding Algorithm pick bundles from top to bottom based on
throughput/msgRate, bundles with low throughput/msgRate are rarely be selected
for shedding. So there is no need to update these bundleData to zk frequently.**
+
+
+# Goals
+
+Filter out bundles with low throughput/msgRate, and do not update these
bundles to zk frequently to reduce the pressure on zk.
+
+
+# High Level Design
+
+Filter out bundles with low throughput/msgRate when leader update bundleData
to zk.
+
+
+# Detailed Design
+
+## Design & Implementation Details
+Add throughput/msg-based bundle report control in the `ExtensibleLoadManager`
and `ModularLoadManagerImpl`.
+
+### Configuration
+
+add configuration:
+```
+ @FieldContext(
+ dynamic = true,
+ category = CATEGORY_LOAD_BALANCER,
+ doc = "minimum throughput in of bundle to be considered for
updating data in metadata store"
+ )
+ private int loadBalancerBundleThroughputThresholdInByte = 0;
+
+ @FieldContext(
+ dynamic = true,
+ category = CATEGORY_LOAD_BALANCER,
+ doc = "minimum message rate in of bundle to be considered for
updating data in metadata store"
+ )
+ private int loadBalancerBundleMsgThreshold = 0;
+```
+if the value of `loadBalancerBundleThroughputThresholdInByte` or
`loadBalancerBundleMsgThreshold` is 0, it means that the corresponding filter
is disabled.
Review Comment:
Maybe we don't need to say it is enabled or disabled. If the throughput or
message rate of a bundle is greater than the minimum throughput or message in
the load report will be included in the bundle load report.
##########
pip/pip-294.md:
##########
@@ -0,0 +1,56 @@
+# Background knowledge
+
+Load balance module in Pulsar broker rely on zk to store and synchronize
metadata about load. Every broker will upload its `LocalBrokerData` to zk, and
leader broker will retrieve all `LocalBrokerData` from zk ,generate all
`BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk.
+
+
+# Motivation
+
+As every bundle in the cluster corresponds to a zk node, it is common that
there are thousands of zk nodes in a cluster, which results into thousands of
read/update operations to zk. This will cause a lot of pressure on zk.
+
+**As All Load Shedding Algorithm pick bundles from top to bottom based on
throughput/msgRate, bundles with low throughput/msgRate are rarely be selected
for shedding. So there is no need to update these bundleData to zk frequently.**
+
+
+# Goals
+
+Filter out bundles with low throughput/msgRate, and do not update these
bundles to zk frequently to reduce the pressure on zk.
+
+
+# High Level Design
+
+Filter out bundles with low throughput/msgRate when leader update bundleData
to zk.
+
+
+# Detailed Design
+
+## Design & Implementation Details
+Add throughput/msg-based bundle report control in the `ExtensibleLoadManager`
and `ModularLoadManagerImpl`.
+
+### Configuration
+
+add configuration:
+```
+ @FieldContext(
+ dynamic = true,
+ category = CATEGORY_LOAD_BALANCER,
+ doc = "minimum throughput in of bundle to be considered for
updating data in metadata store"
Review Comment:
```suggestion
doc = "minimum throughput in bytes per second of the bundle to
be considered in bundle load report"
```
##########
pip/pip-294.md:
##########
@@ -0,0 +1,56 @@
+# Background knowledge
+
+Load balance module in Pulsar broker rely on zk to store and synchronize
metadata about load. Every broker will upload its `LocalBrokerData` to zk, and
leader broker will retrieve all `LocalBrokerData` from zk ,generate all
`BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk.
+
+
+# Motivation
+
+As every bundle in the cluster corresponds to a zk node, it is common that
there are thousands of zk nodes in a cluster, which results into thousands of
read/update operations to zk. This will cause a lot of pressure on zk.
+
+**As All Load Shedding Algorithm pick bundles from top to bottom based on
throughput/msgRate, bundles with low throughput/msgRate are rarely be selected
for shedding. So there is no need to update these bundleData to zk frequently.**
+
+
+# Goals
+
+Filter out bundles with low throughput/msgRate, and do not update these
bundles to zk frequently to reduce the pressure on zk.
+
+
+# High Level Design
+
+Filter out bundles with low throughput/msgRate when leader update bundleData
to zk.
+
+
+# Detailed Design
+
+## Design & Implementation Details
+Add throughput/msg-based bundle report control in the `ExtensibleLoadManager`
and `ModularLoadManagerImpl`.
+
+### Configuration
+
+add configuration:
+```
+ @FieldContext(
+ dynamic = true,
+ category = CATEGORY_LOAD_BALANCER,
+ doc = "minimum throughput in of bundle to be considered for
updating data in metadata store"
+ )
+ private int loadBalancerBundleThroughputThresholdInByte = 0;
+
+ @FieldContext(
+ dynamic = true,
+ category = CATEGORY_LOAD_BALANCER,
+ doc = "minimum message rate in of bundle to be considered for
updating data in metadata store"
+ )
+ private int loadBalancerBundleMsgThreshold = 0;
Review Comment:
```suggestion
private int loadBalancerBundleMinMessageRateInBundleLoadReport = 0;
```
##########
pip/pip-294.md:
##########
@@ -0,0 +1,56 @@
+# Background knowledge
+
+Load balance module in Pulsar broker rely on zk to store and synchronize
metadata about load. Every broker will upload its `LocalBrokerData` to zk, and
leader broker will retrieve all `LocalBrokerData` from zk ,generate all
`BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk.
+
+
+# Motivation
+
+As every bundle in the cluster corresponds to a zk node, it is common that
there are thousands of zk nodes in a cluster, which results into thousands of
read/update operations to zk. This will cause a lot of pressure on zk.
+
+**As All Load Shedding Algorithm pick bundles from top to bottom based on
throughput/msgRate, bundles with low throughput/msgRate are rarely be selected
for shedding. So there is no need to update these bundleData to zk frequently.**
+
+
+# Goals
+
+Filter out bundles with low throughput/msgRate, and do not update these
bundles to zk frequently to reduce the pressure on zk.
+
+
+# High Level Design
+
+Filter out bundles with low throughput/msgRate when leader update bundleData
to zk.
+
+
+# Detailed Design
+
+## Design & Implementation Details
+Add throughput/msg-based bundle report control in the `ExtensibleLoadManager`
and `ModularLoadManagerImpl`.
Review Comment:
Do we have any bundle report data expiration mechanism? For example, if one
bundle with 100MB/s, suddenly drops down to 1MB/s, when the reported data
`100MB/s` will be expired?
It's better to have a description in the proposal to let everyone know the
details of some corner cases handling.
##########
pip/pip-294.md:
##########
@@ -0,0 +1,56 @@
+# Background knowledge
+
+Load balance module in Pulsar broker rely on zk to store and synchronize
metadata about load. Every broker will upload its `LocalBrokerData` to zk, and
leader broker will retrieve all `LocalBrokerData` from zk ,generate all
`BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk.
+
+
+# Motivation
+
+As every bundle in the cluster corresponds to a zk node, it is common that
there are thousands of zk nodes in a cluster, which results into thousands of
read/update operations to zk. This will cause a lot of pressure on zk.
+
+**As All Load Shedding Algorithm pick bundles from top to bottom based on
throughput/msgRate, bundles with low throughput/msgRate are rarely be selected
for shedding. So there is no need to update these bundleData to zk frequently.**
+
+
+# Goals
+
+Filter out bundles with low throughput/msgRate, and do not update these
bundles to zk frequently to reduce the pressure on zk.
+
+
+# High Level Design
+
+Filter out bundles with low throughput/msgRate when leader update bundleData
to zk.
+
+
+# Detailed Design
+
+## Design & Implementation Details
+Add throughput/msg-based bundle report control in the `ExtensibleLoadManager`
and `ModularLoadManagerImpl`.
+
+### Configuration
+
+add configuration:
+```
+ @FieldContext(
+ dynamic = true,
+ category = CATEGORY_LOAD_BALANCER,
+ doc = "minimum throughput in of bundle to be considered for
updating data in metadata store"
+ )
+ private int loadBalancerBundleThroughputThresholdInByte = 0;
+
+ @FieldContext(
+ dynamic = true,
+ category = CATEGORY_LOAD_BALANCER,
+ doc = "minimum message rate in of bundle to be considered for
updating data in metadata store"
Review Comment:
```suggestion
doc = "minimum message rate in per second of the bundle to be
considered in bundle load report"
```
##########
pip/pip-294.md:
##########
@@ -0,0 +1,56 @@
+# Background knowledge
+
+Load balance module in Pulsar broker rely on zk to store and synchronize
metadata about load. Every broker will upload its `LocalBrokerData` to zk, and
leader broker will retrieve all `LocalBrokerData` from zk ,generate all
`BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk.
+
+
+# Motivation
+
+As every bundle in the cluster corresponds to a zk node, it is common that
there are thousands of zk nodes in a cluster, which results into thousands of
read/update operations to zk. This will cause a lot of pressure on zk.
+
+**As All Load Shedding Algorithm pick bundles from top to bottom based on
throughput/msgRate, bundles with low throughput/msgRate are rarely be selected
for shedding. So there is no need to update these bundleData to zk frequently.**
+
+
+# Goals
+
+Filter out bundles with low throughput/msgRate, and do not update these
bundles to zk frequently to reduce the pressure on zk.
Review Comment:
```suggestion
Reduce the load report size by excluding the bundles with low
throughput/msgRate in the bundle load report.
```
##########
pip/pip-294.md:
##########
@@ -0,0 +1,56 @@
+# Background knowledge
+
+Load balance module in Pulsar broker rely on zk to store and synchronize
metadata about load. Every broker will upload its `LocalBrokerData` to zk, and
leader broker will retrieve all `LocalBrokerData` from zk ,generate all
`BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk.
+
+
+# Motivation
+
+As every bundle in the cluster corresponds to a zk node, it is common that
there are thousands of zk nodes in a cluster, which results into thousands of
read/update operations to zk. This will cause a lot of pressure on zk.
+
+**As All Load Shedding Algorithm pick bundles from top to bottom based on
throughput/msgRate, bundles with low throughput/msgRate are rarely be selected
for shedding. So there is no need to update these bundleData to zk frequently.**
+
+
+# Goals
+
+Filter out bundles with low throughput/msgRate, and do not update these
bundles to zk frequently to reduce the pressure on zk.
+
+
+# High Level Design
+
+Filter out bundles with low throughput/msgRate when leader update bundleData
to zk.
+
+
+# Detailed Design
+
+## Design & Implementation Details
+Add throughput/msg-based bundle report control in the `ExtensibleLoadManager`
and `ModularLoadManagerImpl`.
+
+### Configuration
+
+add configuration:
+```
+ @FieldContext(
+ dynamic = true,
+ category = CATEGORY_LOAD_BALANCER,
+ doc = "minimum throughput in of bundle to be considered for
updating data in metadata store"
+ )
+ private int loadBalancerBundleThroughputThresholdInByte = 0;
+
+ @FieldContext(
+ dynamic = true,
+ category = CATEGORY_LOAD_BALANCER,
+ doc = "minimum message rate in of bundle to be considered for
updating data in metadata store"
Review Comment:
Only rate in will be considered?
##########
pip/pip-294.md:
##########
@@ -0,0 +1,56 @@
+# Background knowledge
+
+Load balance module in Pulsar broker rely on zk to store and synchronize
metadata about load. Every broker will upload its `LocalBrokerData` to zk, and
leader broker will retrieve all `LocalBrokerData` from zk ,generate all
`BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk.
+
+
+# Motivation
+
+As every bundle in the cluster corresponds to a zk node, it is common that
there are thousands of zk nodes in a cluster, which results into thousands of
read/update operations to zk. This will cause a lot of pressure on zk.
+
+**As All Load Shedding Algorithm pick bundles from top to bottom based on
throughput/msgRate, bundles with low throughput/msgRate are rarely be selected
for shedding. So there is no need to update these bundleData to zk frequently.**
+
+
+# Goals
+
+Filter out bundles with low throughput/msgRate, and do not update these
bundles to zk frequently to reduce the pressure on zk.
+
+
+# High Level Design
+
+Filter out bundles with low throughput/msgRate when leader update bundleData
to zk.
+
+
+# Detailed Design
+
+## Design & Implementation Details
+Add throughput/msg-based bundle report control in the `ExtensibleLoadManager`
and `ModularLoadManagerImpl`.
+
+### Configuration
+
+add configuration:
+```
+ @FieldContext(
+ dynamic = true,
+ category = CATEGORY_LOAD_BALANCER,
+ doc = "minimum throughput in of bundle to be considered for
updating data in metadata store"
Review Comment:
It should be throughput in + throughput out, right? Or only throughput in
##########
pip/pip-294.md:
##########
@@ -0,0 +1,56 @@
+# Background knowledge
+
+Load balance module in Pulsar broker rely on zk to store and synchronize
metadata about load. Every broker will upload its `LocalBrokerData` to zk, and
leader broker will retrieve all `LocalBrokerData` from zk ,generate all
`BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk.
Review Comment:
And it's better also to add the existing configuration
`loadBalancerMaxNumberOfBundlesInBundleLoadReport` here.
##########
pip/pip-294.md:
##########
@@ -0,0 +1,56 @@
+# Background knowledge
+
+Load balance module in Pulsar broker rely on zk to store and synchronize
metadata about load. Every broker will upload its `LocalBrokerData` to zk, and
leader broker will retrieve all `LocalBrokerData` from zk ,generate all
`BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk.
+
+
+# Motivation
+
+As every bundle in the cluster corresponds to a zk node, it is common that
there are thousands of zk nodes in a cluster, which results into thousands of
read/update operations to zk. This will cause a lot of pressure on zk.
+
+**As All Load Shedding Algorithm pick bundles from top to bottom based on
throughput/msgRate, bundles with low throughput/msgRate are rarely be selected
for shedding. So there is no need to update these bundleData to zk frequently.**
+
+
+# Goals
+
+Filter out bundles with low throughput/msgRate, and do not update these
bundles to zk frequently to reduce the pressure on zk.
Review Comment:
Just to avoid the confusion of "reduce the frequency" or "stop to report".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
