This is an automated email from the ASF dual-hosted git repository.
kwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 9f8adc5962f [improve] [pip] PIP-358: let resource weight work for
OverloadShedder, LeastLongTermMessageRate, ModularLoadManagerImpl. (#22889)
9f8adc5962f is described below
commit 9f8adc5962f2733bd192ebd3b58978ed9cebab14
Author: Wenzhi Feng <[email protected]>
AuthorDate: Mon Jun 17 12:43:23 2024 +0800
[improve] [pip] PIP-358: let resource weight work for OverloadShedder,
LeastLongTermMessageRate, ModularLoadManagerImpl. (#22889)
Implementation PR: https://github.com/apache/pulsar/pull/22888
### Motivation
Initially, we introduce `loadBalancerCPUResourceWeight`,
`loadBalancerBandwidthInResourceWeight`,
`loadBalancerBandwidthOutResourceWeight`, `loadBalancerMemoryResourceWeight`,
`loadBalancerDirectMemoryResourceWeight` in `ThresholdShedder` to control the
resource weight for different resources when calculating the load of the broker.
Then we let it work for `LeastResourceUsageWithWeight` for better bundle
placement policy.
But https://github.com/apache/pulsar/pull/19559 and
https://github.com/apache/pulsar/pull/21168 have point out that the actual load
of the broker is not related to the memory usage and direct memory usage, thus
we have changed the default value of `loadBalancerMemoryResourceWeight`,
`loadBalancerDirectMemoryResourceWeight` to 0.0.
There are still some places where memory usage and direct memory usage are
used to calculate the load of the broker, such as `OverloadShedder`,
`LeastLongTermMessageRate`, `ModularLoadManagerImpl`. We should let the
resource weight work for these places so that we can set the resource weight to
0.0 to avoid the impact of memory usage and direct memory usage on the load of
the broker.
### Modifications
- Let resource weight work for `OverloadShedder`,
`LeastLongTermMessageRate`, `ModularLoadManagerImpl`.
---
pip/pip-358.md | 38 ++++++++++++++++++++++++++++++++++++++
1 file changed, 38 insertions(+)
diff --git a/pip/pip-358.md b/pip/pip-358.md
new file mode 100644
index 00000000000..cd539730985
--- /dev/null
+++ b/pip/pip-358.md
@@ -0,0 +1,38 @@
+
+# PIP-358: let resource weight work for OverloadShedder,
LeastLongTermMessageRate, ModularLoadManagerImpl.
+
+# Background knowledge
+
+Initially, we introduce `loadBalancerCPUResourceWeight`,
`loadBalancerBandwidthInResourceWeight`,
`loadBalancerBandwidthOutResourceWeight`,
+`loadBalancerMemoryResourceWeight`, `loadBalancerDirectMemoryResourceWeight`
in `ThresholdShedder` to control the resource weight for
+different resources when calculating the load of the broker.
+Then we let it work for `LeastResourceUsageWithWeight` for better bundle
placement policy.
+
+But https://github.com/apache/pulsar/pull/19559 and
https://github.com/apache/pulsar/pull/21168 have pointed out that the actual
load
+of the broker is not related to the memory usage and direct memory usage, thus
we have changed the default value of
+`loadBalancerMemoryResourceWeight`, `loadBalancerDirectMemoryResourceWeight`
to 0.0.
+
+There are still some places where memory usage and direct memory usage are
used to calculate the load of the broker, such as
+`OverloadShedder`, `LeastLongTermMessageRate`, `ModularLoadManagerImpl`. We
should let the resource weight work for these places
+so that we can set the resource weight to 0.0 to avoid the impact of memory
usage and direct memory usage on the load of the broker.
+
+# Motivation
+
+The actual load of the broker is not related to the memory usage and direct
memory usage, thus we should let the resource weight work for
+`OverloadShedder`, `LeastLongTermMessageRate`, `ModularLoadManagerImpl` so
that we can set the resource weight to 0.0 to avoid the impact of
+memory usage and direct memory usage on the load of the broker.
+
+
+# Detailed Design
+
+Let resource weight work for `OverloadShedder`, `LeastLongTermMessageRate`,
`ModularLoadManagerImpl`.
+- For `OverloadShedder`, `LeastLongTermMessageRate`, we replace
`getMaxResourceUsage()` with `getMaxResourceUsageWithWeight()` in the
calculation of the load of the broker.
+- For `ModularLoadManagerImpl`, we replace `getMaxResourceUsage()` with
`getMaxResourceUsageWithWeight()` when checking if the broker is overloaded and
decide whether to update the broker data to metadata store.
+
+# Backward & Forward Compatibility
+
+
+# Links
+
+* Mailing List discussion thread:
https://lists.apache.org/thread/lj34s3vmjbzlwmy8d66d0bsb25vnq9ky
+* Mailing List voting thread:
https://lists.apache.org/thread/b7dzm0yz6l40pkxmxhto5mro7brmz57r
