Anonymitaet commented on code in PR #644:
URL: https://github.com/apache/pulsar-site/pull/644#discussion_r1266616582
##########
docs/concepts-broker-load-balancing-concepts.md:
##########
@@ -4,4 +4,554 @@ title: Concepts
sidebar_label: "Concepts"
---
-WIP. Stay tuned!
\ No newline at end of file
+````mdx-code-block
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+````
+
+Pulsar provides robust support for load balancing to ensure efficient
utilization of resources across Pulsar clusters. Load balancing in Pulsar
involves distributing messages and partitions evenly among brokers and
consumers to prevent hotspots and optimize performance.
+
+Before getting started with load balancing, it's important to review the key
components to ensure that resources are utilized efficiently and varying
workloads can be handled by the system effectively.
+
+## Brokers
+
+In a Pulsar cluster, [brokers](./reference-terminology.md#broker) are
responsible for serving messages for different topics and partitions. Broker
load balancing ensures that each broker handles a proportional share of the
load.
+
+## Producers
+
+[Producers](./reference-terminology.md#producer) in Pulsar are responsible for
publishing messages to topics. Pulsar clients (producers) connect to brokers to
publish messages. Producer load balancing (i.e., connection pooling mechanism
in Pulsar) ensures that producers are distributed across brokers to avoid
overwhelming a single broker with too many connections.
+
+## Consumers
+
+[Consumers](./reference-terminology.md#consumer) in Pulsar are responsible for
consuming messages from topics. Depending on how consumer load balancing is
configured (i.e., using exclusive or shared consumers or auto-rebalancing), you
can ensure even load distribution.
+
+## Topics
+
+[Topics](./reference-terminology.md#topic) are the basic units for clients to
publish and consume messages. Related topics are logically grouped into a
namespace. To efficiently manage metadata and keep track of all of them moving
through the system, Pulsar uses a strategy of grouping topics by partitioning
on a namespace to create topic bundles.
+
+
+
+## Bundles
+
+[Bundles](./reference-terminology.md#namespace-bundle) represent a range of
partitions for a particular namespace in Pulsar, comprising a portion of the
overall hash range of the namespace.
+
+Bundle is introduced in Pulsar to represent a middle-layer group. Each bundle
is an **assignment unit**, which means topics are assigned to brokers at the
**bundle** level rather than the topic level.
+
+## Broker load balancing
+
+The broker load balancer component is like a "traffic cop" sitting between
clients and brokers. It balances topic sessions across brokers based on dynamic
load data, such as broker resource usage (e.g., CPU, memory, network IO) and
topic/bundle loads (e.g., throughput).
+
+When properly balanced, the brokers can handle increased traffic and ensure
that the system can scale seamlessly to accommodate growing workloads. Load
balancing helps prevent bottlenecks and ensures that the resources of the
cluster are utilized optimally, leading to better throughput and reduced
message processing latency.
+
+
+
+## Topic bundling
+
+Topic bundling refers to the process of grouping topics into bundles. Pulsar
organizes topics into bundles within a namespace. Each bundle is a range of
partitions, and Pulsar can automatically distribute these bundles across
brokers to achieve load balancing. This allows the cluster to scale more
efficiently as brokers can independently manage their assigned bundles.
+
+For example,
+
+- Topic load statistics (e.g., message rates) are aggregated at the **bundle**
layer, which reduces the cardinality of load samples to monitor.
+
+- For dynamic topic-broker assignments, Pulsar persists these mappings at the
**bundle **level, which decreases the space for storing dynamic topic-broker
ownerships.
+
+Pulsar allows you to dynamically scale the number of brokers, producers, and
consumers to adapt to changing workloads. As brokers are added or removed,
Pulsar handles the redistribution of partitions and bundles automatically.
+
+### Workflow
+
+Below is the workflow for grouping topics into bundles.
+
+#### Step 1: shard namespaces into bundles
+
+Internally, when a namespace is created, the namespace is sharded into a list
of bundles.
+
+#### Step 2: assign topics to bundles
+
+When a topic is created or looked up for pub/sub sessions, brokers map the
topic to a particular bundle by taking the hash of the topic name (for example,
hash("my-topic") = 0x0000000F) and checking in which bundle the hash falls.
+
+Here "topic" means either a **non-partitioned topic** or **one partition of a
partitioned topic**. For partitioned topics, Pulsar internally considers
partitions as separate topics, hence different partitions can be assigned to
different bundles and brokers.
+
+
+
+## Bundle assignment
+
+Bundle assignment refers to assigning bundles to brokers dynamically based on
changing conditions.
+
+For example, based on broker resource usage (e.g., CPU, memory, network IO)
and bundle loads (e.g., throughput), a bundle is dynamically assigned to a
particular broker. Each bundle is independent of the others and thus is
independently assigned to different brokers. Each broker takes ownership of a
bundle (aka, a subset of the topics for a namespace).
+
+Bundle assignment plays a crucial role in achieving efficient load
distribution and scalability within a Pulsar cluster. The purpose of bundle
assignments is to ensure balanced resource utilization and facilitate dynamic
scaling within the Pulsar architecture.
+
+### Workflow
+
+Below is the workflow for dynamic bundle assignment.
+
+#### Step 1: assign bundles to brokers dynamically
+
+When a client starts using new topics (bundles) that are not assigned to any
broker, a process is triggered to choose the best-suited broker to acquire
ownership of these bundles according to the load conditions.
+
+#### Step 2: reassign bundles to other brokers (optional)
+
+If a broker owning a bundle crashes, the bundle (topic) is reassigned to
another available broker.
+
+
+
+To discover the current bundle-broker ownership for a given topic, Pulsar uses
a server-side discovery mechanism that redirects clients to the owner brokers'
URLs. This discovery logic requires:
+
+- Bundle key ranges for a given namespace, to map a topic to a bundle.
+
+- Bundle-broker ownership mapping, to direct the client to the current owner
or to trigger a new ownership acquisition in case there is no broker assigned.
+
+- All bundle ranges and broker-bundle ownership mappings are stored in a
metadata space, and brokers look up them when clients try to discover owner
brokers. For performance reasons, these data are cached at the broker in-memory
layer too.
+
+## Bundle splitting
+
+Bundle splitting refers to the process of identifying and splitting overloaded
bundles, which helps reduce hot spots, achieve more granular load balancing,
improve resource utilization, and enable finer-grained horizontal scaling
within the Pulsar cluster.
+
+The bundle splitting process involves breaking down the original bundle into
smaller bundles, each containing a subset of the original partitions. This
allows for better distribution of the message and processing load across
brokers in the cluster.
+
+You can split bundles in the following ways:
+
+- Automatic: enable Pulsar's automatic bundle splitting process when a
namespace has a significant increase in workload or the number of partitions
exceeds the optimal capacity for a single bundle.
+
+- Manual: trigger bundle splitting manually, to divide an existing bundle into
multiple smaller bundles.
+
+Bundle splitting methods|Definition|When to use
+|---|---|---
+Automatic|Bundles are split automatically based on different [bundle splitting
algorithms](#bundle-splitting-algorithms). | Automatic bundle splitting is most
commonly used.<br/><br/>You can use this method in various scenarios, such as
when a bundle remains hot for a long time.
+Manual|Bundles are split manually based on specified positions.|Manual bundle
splitting serves as a supplementary approach to automatic bundle
splitting.<br/><br/>You can use this method in various scenarios, such as:
<br/><br/> - If automatic bundle splitting is enabled, but there are still
bundles that remain hot for a long time. <br/><br/> - If you want to split
bundles and redistribute traffic evenly before having any broker overloaded.
+
+### Workflow
+
+Below is the workflow for splitting bundles automaticaly or manually.
+
+````mdx-code-block
+<Tabs groupId="bundle-splitting-workflow"
+ defaultValue="Automatic bundle splitting"
+ values={[{"label":"Automatic bundle splitting","value":"Automatic bundle
splitting"},{"label":"Manual bundle splitting","value":"Manual bundle
splitting"}]}>
+<TabItem value="Automatic bundle splitting">
+
+#### Step 1: find target bundles
+
+If the auto bundle split is enabled,
+
+- For the modular load balancer, the leader broker will check if any bundle's
load is beyond the threshold.
+
+- For the extensible load balancer, the load manager will check the bundle's
load in each owner broker.
+
+Bundle splitting threshold can be set based on various conditions. Any
existing bundle that exceeds any of the thresholds is a candidate to be split.
The load balancer assigns the newly split bundles to other brokers, to
facilitate the traffic distribution.
+
+For how to enable bundle split and set bundle split thresholds automatically,
see TBD (the docs is WIP, stay tuned!).
Review Comment:
For reviewers:
I write this on purpose. Reasons:
- Give users a positive expectation
- Call for a targeted list of early adopters interested in this new feature
- Improve SEO by adding relevant keywords
> see TBD (the docs is WIP, stay tuned!).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
