Anonymitaet commented on code in PR #662: URL: https://github.com/apache/pulsar-site/pull/662#discussion_r1281695932
########## docs/develop-load-manager.md: ########## @@ -207,3 +103,216 @@ then all machines are approximately overloaded. In the case in which a broker's threshold, that broker is not considered for bundle assignment. If all brokers are overloaded, the bundle is randomly assigned. +## Enablement + +You can enable a simple, modular, or extensible load manager by following the steps below. + +### Enable simple load manager + +You can enable the simple load manager using one of the following methods: + +- Method 1 + + Update the value of [loadManagerClassName](https://github.com/apache/pulsar/blob/782e91fe327efe2c9c9107d6c679c2837d43935b/conf/broker.conf#L1309) to `org.apache.pulsar.broker.loadbalance.impl.SimpleLoadManagerImpl` in `conf/broker.conf`. + +- Method 2 + + Use the `pulsar-admin` tool. + + ```shell + pulsar-admin brokers update-dynamic-config \ + --config loadManagerClassName \ + --value org.apache.pulsar.broker.loadbalance.impl.SimpleLoadManagerImpl + ``` + + :::note + + Any mistakes in specifying the load manager will cause Pulsar to default to `SimpleLoadManagerImpl`. + + ::: + +### Enable modular load manager + +You can enable the modular load manager using one of the following methods: + +- Method 1 + + Update the value of [loadManagerClassName](https://github.com/apache/pulsar/blob/782e91fe327efe2c9c9107d6c679c2837d43935b/conf/broker.conf#L1309) to `org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl` in `conf/broker.conf`. + +- Method 2 + + Use the `pulsar-admin` tool. + + ```shell + pulsar-admin brokers update-dynamic-config \ + --config loadManagerClassName \ + --value org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl + ``` + + :::note + + Any mistakes in specifying the load manager will cause Pulsar to default to `SimpleLoadManagerImpl`. + + ::: + +### Enable extensible load manager + +You can enable the extensible load manager by updating the value of [loadManagerClassName](https://github.com/apache/pulsar/blob/782e91fe327efe2c9c9107d6c679c2837d43935b/conf/broker.conf#L1309) to `org.apache.pulsar.broker.loadbalance.impl.ExtensibleLoadManagerImpl` in `conf/broker.conf`. + +:::note + +The pulsar-admin tool is not supported for enabling the extensible load manager. + +::: + +## Verification + +If you want to verify which load manager is used, follow the steps below. + +### Step 1: check loadManagerClassName + +You can use the `pulsar-admin` tool to examine the `loadManagerClassName` element. + +**Input** + +```bash +bin/pulsar-admin brokers get-all-dynamic-config +``` + +**Output** + +```bash +{ +"loadManagerClassName" : "org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl" +} +``` + +If there is no `loadManagerClassName` element, then the value of [loadManagerClassName](https://github.com/apache/pulsar/blob/782e91fe327efe2c9c9107d6c679c2837d43935b/conf/broker.conf#L1309) in the `conf/broker.conf` file is used. + +### Step 2: verify load manager (optional) + +To double-check which load manager is used, you can [check the ZooKeeper load report](#method-1-check-zookeeper-load-report) or [check monitor-brokers output](#method-2-check-monitor-brokers-output). + +#### Method 1: check ZooKeeper load report + +Check the ZooKeeper load report. The load report in `/loadbalance/brokers/...`. + +```bash +tbd +``` + +Different load managers have different ZooKeeper load reports. + +- [Simple load manager](./concepts-broker-load-balancing-types.md) + + ```json + { + "systemResourceUsage": { + "bandwidthIn": { + "limit": 10240000.0, + "usage": 0.0 + }, + "bandwidthOut": { + "limit": 10240000.0, + "usage": 0.0 + }, + "cpu": { + "limit": 2400.0, + "usage": 0.0 + }, + "directMemory": { + "limit": 16384.0, + "usage": 1.0 + }, + "memory": { + "limit": 8192.0, + "usage": 3903.0 + } + } + } + ``` + +- [Modular load manager](./concepts-broker-load-balancing-types.md) + + Modular load manager has different ZooKeeper load report from that of the simple load manager. For example, the `systemResourceUsage` sub-elements (`bandwidthIn`, `bandwidthOut`, etc.) are now all at the top level. + + ```json + { + "bandwidthIn": { + "limit": 10240000.0, + "usage": 4.256510416666667 + }, + "bandwidthOut": { + "limit": 10240000.0, + "usage": 5.287239583333333 + }, + "bundles": [], + "cpu": { + "limit": 2400.0, + "usage": 5.7353247655435915 + }, + "directMemory": { + "limit": 16384.0, + "usage": 1.0 + } + } + ``` + +- Extensible load manager + + :::note + + [Extensible load manager](./concepts-broker-load-balancing-types.md) does not have the ZooKeepr load report because its internal stats are stored in system topics rather than ZooKeeper. + + ::: + + +#### Method 2: check monitor-brokers output + +You can use the [pulsar-perf tool](reference-cli-tools.md) to start a broker monitor. + +```bash +pulsar-perf monitor-brokers --connect-string <zookeeper host:port> +```` + +Different load managers have different outputs. This output is the same as the output of the [Method 1: check ZooKeeper load report](#method-1-check-zookeeper-load-report), but it is well formatted for better readability. + +- Simple load manager + + ```bash + =================================================================================================================== + ||COUNT |TOPIC |BUNDLE |PRODUCER |CONSUMER |BUNDLE + |BUNDLE - || + || |4 |4 |0 |2 |0 |0 || + ||RAW SYSTEM |CPU % |MEMORY % |DIRECT % |BW IN % |BW OUT % |MAX % || + || |0.25 |47.94 |0.01 |0.00 |0.00 |47.94 || + ||ALLOC SYSTEM |CPU % |MEMORY % |DIRECT % |BW IN % |BW OUT % |MAX % || + || |0.20 |1.89 | |1.27 |3.21 |3.21 || + ||RAW MSG |MSG/S IN |MSG/S OUT |TOTAL |KB/S IN |KB/S OUT |TOTAL || + || |0.00 |0.00 |0.00 |0.01 |0.01 |0.01 || + ||ALLOC MSG |MSG/S IN |MSG/S OUT |TOTAL |KB/S IN |KB/S OUT |TOTAL || + || |54.84 |134.48 |189.31 |126.54 |320.96 |447.50 || + =================================================================================================================== + ``` + + - Modular load manager + + ```bash + =================================================================================================================== + ||SYSTEM |CPU % |MEMORY % |DIRECT % |BW IN % |BW OUT % |MAX % || + || |0.00 |48.33 |0.01 |0.00 |0.00 |48.33 || + ||COUNT |TOPIC |BUNDLE |PRODUCER |CONSUMER |BUNDLE + |BUNDLE - || + || |4 |4 |0 |2 |4 |0 || + ||LATEST |MSG/S IN |MSG/S OUT |TOTAL |KB/S IN |KB/S OUT |TOTAL || + || |0.00 |0.00 |0.00 |0.00 |0.00 |0.00 || + ||SHORT |MSG/S IN |MSG/S OUT |TOTAL |KB/S IN |KB/S OUT |TOTAL || + || |0.00 |0.00 |0.00 |0.00 |0.00 |0.00 || + ||LONG |MSG/S IN |MSG/S OUT |TOTAL |KB/S IN |KB/S OUT |TOTAL || + || |0.00 |0.00 |0.00 |0.00 |0.00 |0.00 || + =================================================================================================================== + ``` + + - Extensible load manager + + ```bash + tbd Review Comment: @Demogorgon314 as we discussed just now, can you provide technical input for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
