This is an automated email from the ASF dual-hosted git repository.
liuyu pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new a8e46d0b3e4 [feat][doc] add overview for broker load balancing (#621)
a8e46d0b3e4 is described below
commit a8e46d0b3e4c8d92ac980125be1d826ad17a72a0
Author: Anonymitaet <[email protected]>
AuthorDate: Wed Jun 28 20:32:17 2023 +0800
[feat][doc] add overview for broker load balancing (#621)
---
docs/concepts-broker-load-balancing-concepts.md | 7 ++++++
docs/concepts-broker-load-balancing-overview.md | 32 +++++++++++++++++++++++++
sidebars.json | 8 +++++++
3 files changed, 47 insertions(+)
diff --git a/docs/concepts-broker-load-balancing-concepts.md
b/docs/concepts-broker-load-balancing-concepts.md
new file mode 100644
index 00000000000..a618d90c479
--- /dev/null
+++ b/docs/concepts-broker-load-balancing-concepts.md
@@ -0,0 +1,7 @@
+---
+id: concepts-broker-load-balancing-concepts
+title: Concepts
+sidebar_label: "Concepts"
+---
+
+WIP. Stay tuned!
\ No newline at end of file
diff --git a/docs/concepts-broker-load-balancing-overview.md
b/docs/concepts-broker-load-balancing-overview.md
new file mode 100644
index 00000000000..91283f2943d
--- /dev/null
+++ b/docs/concepts-broker-load-balancing-overview.md
@@ -0,0 +1,32 @@
+---
+id: concepts-broker-load-balancing-overview
+title: Overview
+sidebar_label: "Overview"
+---
+
+
+## Challenges of load balancing in distributed streaming systems
+
+Like other distributed systems, load balancing is important in messaging and
streaming systems. Without it, load imbalance can cause hot-spot brokers,
resulting in performance degradation, cluster unavailability, and wasted broker
resources.
+
+Due to the unpredictable topic volume and physical distance among distributed
brokers, it is not easy to dynamically distribute message loads among brokers.
It requires the system to continuously monitor and route message loads based on
changing conditions without compromising system performance. For example:
+
+- When topics receive high traffic, exhausting CPU or memory resources on
particular brokers, the cluster offloads the overloaded brokers and
redistributes the load to other brokers.
+
+- When brokers experience low traffic, become idle, or are added or removed,
the cluster rebalances the load to avoid wasting resources.
+
+- When topics are redistributed to other brokers, the cluster ensures the
topics are instantaneously available to clients. The topics continue to
guarantee the system performance, such as persistence,
[ordering](./concepts-messaging.md#ordering-guarantee),
[deduplication](./concepts-messaging.md#message-deduplication), [subscription
type](./concepts-messaging.md#subscription-types), etc.
+
+## Load balancing in Pulsar
+
+Because Pulsar uses a [segment-centric
architecture](./concepts-architecture-overview.md) and separates the message
serving and storage layer, it is designed to benefit load balancing.
+
+- At the persistence layer ([BookKeeper](https://bookkeeper.apache.org/)),
message segments in topics are balanced across all the bookies in the cluster.
When an individual bookie runs out of storage capacity, the rest segments are
loaded into the available bookies.
+
+- At the serving layer
([broker](./concepts-architecture-overview.md#brokers)), topic rearrangement
(balance) is seamless. Brokers do not need to copy messages from one broker to
another when rebalancing topics among brokers. Instead, the current owner
broker temporarily closes the topic and client sessions and transfers the
ownership to the selected broker. Then, the selected broker takes ownership of
the topic and opens the topic sessions to the clients.
+
+Pulsar uses automatic broker load balancing to monitor the brokers' load
internally and then dynamically balances topic sessions according to the load
across all available brokers as evenly, dynamically, and flexibly as possible.
Consequently, it improves performance, availability, and usage of resources.
+
+## Related topics
+
+- To learn essential fundamentals, see [broker load balancing |
concepts](./concepts-broker-load-balancing-concepts.md).
diff --git a/sidebars.json b/sidebars.json
index 841b038f35c..a16ddfb0b70 100644
--- a/sidebars.json
+++ b/sidebars.json
@@ -26,6 +26,14 @@
"concepts-messaging",
"concepts-architecture-overview",
"concepts-clients",
+ {
+ "type": "category",
+ "label": "Broker load balancing",
+ "items": [
+ "concepts-broker-load-balancing-overview",
+ "concepts-broker-load-balancing-concepts"
+ ]
+ },
"concepts-replication",
"concepts-cluster-level-failover",
"concepts-multi-tenancy",
