This is an automated email from the ASF dual-hosted git repository.
liuyu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new ecff9d0 [website][upgrade]feat: docs migration - 2.7.1 /
administration (#12609)
ecff9d0 is described below
commit ecff9d02809f360eab452149d3c0e5acafbf3b6c
Author: Li Li <[email protected]>
AuthorDate: Fri Nov 5 15:29:37 2021 +0800
[website][upgrade]feat: docs migration - 2.7.1 / administration (#12609)
* [website][upgrade]feat: docs migration - 2.7.1 / deploy
Signed-off-by: LiLi <[email protected]>
* [website][upgrade]feat: docs migration - 2.7.1 / administration
Signed-off-by: LiLi <[email protected]>
---
.../version-2.7.1/administration-geo.md | 218 ++++++++++++
.../version-2.7.1/administration-isolation.md | 129 +++++++
.../version-2.7.1/administration-load-balance.md | 204 +++++++++++
.../version-2.7.1/administration-proxy.md | 90 +++++
.../version-2.7.1/administration-pulsar-manager.md | 209 +++++++++++
.../version-2.7.1/administration-stats.md | 68 ++++
.../version-2.7.1/administration-upgrade.md | 172 ++++++++++
.../version-2.7.1/administration-zk-bk.md | 381 +++++++++++++++++++++
.../versioned_sidebars/version-2.7.1-sidebars.json | 38 ++
9 files changed, 1509 insertions(+)
diff --git
a/site2/website-next/versioned_docs/version-2.7.1/administration-geo.md
b/site2/website-next/versioned_docs/version-2.7.1/administration-geo.md
new file mode 100644
index 0000000..37b52e2
--- /dev/null
+++ b/site2/website-next/versioned_docs/version-2.7.1/administration-geo.md
@@ -0,0 +1,218 @@
+---
+id: administration-geo
+title: Pulsar geo-replication
+sidebar_label: "Geo-replication"
+original_id: administration-geo
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+*Geo-replication* is the replication of persistently stored message data
across multiple clusters of a Pulsar instance.
+
+## How geo-replication works
+
+The diagram below illustrates the process of geo-replication across Pulsar
clusters:
+
+
+
+In this diagram, whenever **P1**, **P2**, and **P3** producers publish
messages to the **T1** topic on **Cluster-A**, **Cluster-B**, and **Cluster-C**
clusters respectively, those messages are instantly replicated across clusters.
Once the messages are replicated, **C1** and **C2** consumers can consume those
messages from their respective clusters.
+
+Without geo-replication, **C1** and **C2** consumers are not able to consume
messages that **P3** producer publishes.
+
+## Geo-replication and Pulsar properties
+
+You must enable geo-replication on a per-tenant basis in Pulsar. You can
enable geo-replication between clusters only when a tenant is created that
allows access to both clusters.
+
+Although geo-replication must be enabled between two clusters, actually
geo-replication is managed at the namespace level. You must complete the
following tasks to enable geo-replication for a namespace:
+
+* [Enable geo-replication namespaces](#enable-geo-replication-namespaces)
+* Configure that namespace to replicate across two or more provisioned clusters
+
+Any message published on *any* topic in that namespace is replicated to all
clusters in the specified set.
+
+## Local persistence and forwarding
+
+When messages are produced on a Pulsar topic, messages are first persisted in
the local cluster, and then forwarded asynchronously to the remote clusters.
+
+In normal cases, when connectivity issues are none, messages are replicated
immediately, at the same time as they are dispatched to local consumers.
Typically, the network [round-trip
time](https://en.wikipedia.org/wiki/Round-trip_delay_time) (RTT) between the
remote regions defines end-to-end delivery latency.
+
+Applications can create producers and consumers in any of the clusters, even
when the remote clusters are not reachable (like during a network partition).
+
+Producers and consumers can publish messages to and consume messages from any
cluster in a Pulsar instance. However, subscriptions cannot only be local to
the cluster where the subscriptions are created but also can be transferred
between clusters after replicated subscription is enabled. Once replicated
subscription is enabled, you can keep subscription state in synchronization.
Therefore, a topic can be asynchronously replicated across multiple
geographical regions. In case of failover [...]
+
+In the aforementioned example, the **T1** topic is replicated among three
clusters, **Cluster-A**, **Cluster-B**, and **Cluster-C**.
+
+All messages produced in any of the three clusters are delivered to all
subscriptions in other clusters. In this case, **C1** and **C2** consumers
receive all messages that **P1**, **P2**, and **P3** producers publish.
Ordering is still guaranteed on a per-producer basis.
+
+## Configure replication
+
+As stated in [Geo-replication and Pulsar
properties](#geo-replication-and-pulsar-properties) section, geo-replication in
Pulsar is managed at the [tenant](reference-terminology.md#tenant) level.
+
+The following example connects three clusters: **us-east**, **us-west**, and
**us-cent**.
+
+### Connect replication clusters
+
+To replicate data among clusters, you need to configure each cluster to
connect to the other. You can use the
[`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) tool to create
a connection.
+
+**Example**
+
+Suppose that you have 3 replication clusters: `us-west`, `us-cent`, and
`us-east`.
+
+1. Configure the connection from `us-west` to `us-east`.
+
+ Run the following command on `us-west`.
+
+```shell
+
+$ bin/pulsar-admin clusters create \
+ --broker-url pulsar://<DNS-OF-US-EAST>:<PORT> \
+ --url http://<DNS-OF-US-EAST>:<PORT> \
+ us-east
+
+```
+
+:::tip
+
+- If you want to use a secure connection for a cluster, you can use the flags
`--broker-url-secure` and `--url-secure`. For more information, see
[pulsar-admin clusters create](https://pulsar.apache.org/tools/pulsar-admin/).
+- Different clusters may have different authentications. You can use the
authentication flag `--auth-plugin` and `--auth-parameters` together to set
cluster authentication, which overrides `brokerClientAuthenticationPlugin` and
`brokerClientAuthenticationParameters` if `authenticationEnabled` sets to
`true` in `broker.conf` and `standalone.conf`. For more information, see
[authentication and authorization](concepts-authentication).
+
+:::
+
+2. Configure the connection from `us-west` to `us-cent`.
+
+ Run the following command on `us-west`.
+
+```shell
+
+$ bin/pulsar-admin clusters create \
+ --broker-url pulsar://<DNS-OF-US-CENT>:<PORT> \
+ --url http://<DNS-OF-US-CENT>:<PORT> \
+ us-cent
+
+```
+
+3. Run similar commands on `us-east` and `us-cent` to create connections among
clusters.
+
+### Grant permissions to properties
+
+To replicate to a cluster, the tenant needs permission to use that cluster.
You can grant permission to the tenant when you create the tenant or grant
later.
+
+Specify all the intended clusters when you create a tenant:
+
+```shell
+
+$ bin/pulsar-admin tenants create my-tenant \
+ --admin-roles my-admin-role \
+ --allowed-clusters us-west,us-east,us-cent
+
+```
+
+To update permissions of an existing tenant, use `update` instead of `create`.
+
+### Enable geo-replication namespaces
+
+You can create a namespace with the following command sample.
+
+```shell
+
+$ bin/pulsar-admin namespaces create my-tenant/my-namespace
+
+```
+
+Initially, the namespace is not assigned to any cluster. You can assign the
namespace to clusters using the `set-clusters` subcommand:
+
+```shell
+
+$ bin/pulsar-admin namespaces set-clusters my-tenant/my-namespace \
+ --clusters us-west,us-east,us-cent
+
+```
+
+You can change the replication clusters for a namespace at any time, without
disruption to ongoing traffic. Replication channels are immediately set up or
stopped in all clusters as soon as the configuration changes.
+
+### Use topics with geo-replication
+
+Once you create a geo-replication namespace, any topics that producers or
consumers create within that namespace is replicated across clusters.
Typically, each application uses the `serviceUrl` for the local cluster.
+
+#### Selective replication
+
+By default, messages are replicated to all clusters configured for the
namespace. You can restrict replication selectively by specifying a replication
list for a message, and then that message is replicated only to the subset in
the replication list.
+
+The following is an example for the [Java API](client-libraries-java). Note
the use of the `setReplicationClusters` method when you construct the {@inject:
javadoc:Message:/client/org/apache/pulsar/client/api/Message} object:
+
+```java
+
+List<String> restrictReplicationTo = Arrays.asList(
+ "us-west",
+ "us-east"
+);
+
+Producer producer = client.newProducer()
+ .topic("some-topic")
+ .create();
+
+producer.newMessage()
+ .value("my-payload".getBytes())
+ .setReplicationClusters(restrictReplicationTo)
+ .send();
+
+```
+
+#### Topic stats
+
+Topic-specific statistics for geo-replication topics are available via the
[`pulsar-admin`](reference-pulsar-admin) tool and {@inject: rest:REST:/} API:
+
+```shell
+
+$ bin/pulsar-admin persistent stats
persistent://my-tenant/my-namespace/my-topic
+
+```
+
+Each cluster reports its own local stats, including the incoming and outgoing
replication rates and backlogs.
+
+#### Delete a geo-replication topic
+
+Given that geo-replication topics exist in multiple regions, directly deleting
a geo-replication topic is not possible. Instead, you should rely on automatic
topic garbage collection.
+
+In Pulsar, a topic is automatically deleted when the topic meets the following
three conditions:
+- no producers or consumers are connected to it;
+- no subscriptions to it;
+- no more messages are kept for retention.
+For geo-replication topics, each region uses a fault-tolerant mechanism to
decide when deleting the topic locally is safe.
+
+You can explicitly disable topic garbage collection by setting
`brokerDeleteInactiveTopicsEnabled` to `false` in your [broker
configuration](reference-configuration.md#broker).
+
+To delete a geo-replication topic, close all producers and consumers on the
topic, and delete all of its local subscriptions in every replication cluster.
When Pulsar determines that no valid subscription for the topic remains across
the system, it will garbage collect the topic.
+
+## Replicated subscriptions
+
+Pulsar supports replicated subscriptions, so you can keep subscription state
in sync, within a sub-second timeframe, in the context of a topic that is being
asynchronously replicated across multiple geographical regions.
+
+In case of failover, a consumer can restart consuming from the failure point
in a different cluster.
+
+### Enable replicated subscription
+
+Replicated subscription is disabled by default. You can enable replicated
subscription when creating a consumer.
+
+```java
+
+Consumer<String> consumer = client.newConsumer(Schema.STRING)
+ .topic("my-topic")
+ .subscriptionName("my-subscription")
+ .replicateSubscriptionState(true)
+ .subscribe();
+
+```
+
+### Advantages
+
+ * It is easy to implement the logic.
+ * You can choose to enable or disable replicated subscription.
+ * When you enable it, the overhead is low, and it is easy to configure.
+ * When you disable it, the overhead is zero.
+
+### Limitations
+
+When you enable replicated subscription, you're creating a consistent
distributed snapshot to establish an association between message ids from
different clusters. The snapshots are taken periodically. The default value is
`1 second`. It means that a consumer failing over to a different cluster can
potentially receive 1 second of duplicates. You can also configure the
frequency of the snapshot in the `broker.conf` file.
diff --git
a/site2/website-next/versioned_docs/version-2.7.1/administration-isolation.md
b/site2/website-next/versioned_docs/version-2.7.1/administration-isolation.md
new file mode 100644
index 0000000..f57844b
--- /dev/null
+++
b/site2/website-next/versioned_docs/version-2.7.1/administration-isolation.md
@@ -0,0 +1,129 @@
+---
+id: administration-isolation
+title: Pulsar isolation
+sidebar_label: "Pulsar isolation"
+original_id: administration-isolation
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+In an organization, a Pulsar instance provides services to multiple teams.
When organizing the resources across multiple teams, you want to make a
suitable isolation plan to avoid the resource competition between different
teams and applications and provide high-quality messaging service. In this
case, you need to take resource isolation into consideration and weigh your
intended actions against expected and unexpected consequences.
+
+To enforce resource isolation, you can use the Pulsar isolation policy, which
allows you to allocate resources (**broker** and **bookie**) for the namespace.
+
+## Broker isolation
+
+In Pulsar, when namespaces (more specifically, namespace bundles) are assigned
dynamically to brokers, the namespace isolation policy limits the set of
brokers that can be used for assignment. Before topics are assigned to brokers,
you can set the namespace isolation policy with a primary or a secondary regex
to select desired brokers.
+
+You can set a namespace isolation policy for a cluster using one of the
following methods.
+
+<Tabs
+ defaultValue="Admin CLI"
+ values={[
+ {
+ "label": "Admin CLI",
+ "value": "Admin CLI"
+ },
+ {
+ "label": "REST API",
+ "value": "REST API"
+ },
+ {
+ "label": "Java admin API",
+ "value": "Java admin API"
+ }
+]}>
+
+<TabItem value="Admin CLI">
+
+```
+
+pulsar-admin ns-isolation-policy set options
+
+```
+
+For more information about the command `pulsar-admin ns-isolation-policy set
options`, see [here](https://pulsar.apache.org/tools/pulsar-admin/).
+
+**Example**
+
+```shell
+
+bin/pulsar-admin ns-isolation-policy set \
+--auto-failover-policy-type min_available \
+--auto-failover-policy-params min_limit=1,usage_threshold=80 \
+--namespaces my-tenant/my-namespace \
+--primary 10.193.216.* my-cluster policy-name
+
+```
+
+</TabItem>
+<TabItem value="REST API">
+
+[PUT
/admin/v2/namespaces/{tenant}/{namespace}](https://pulsar.apache.org/admin-rest-api/?version=2.7.0&apiversion=v2#operation/createNamespace)
+
+</TabItem>
+<TabItem value="Java admin API">
+
+For how to set namespace isolation policy using Java admin API, see
[here](https://github.com/apache/pulsar/blob/master/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/NamespacesImpl.java#L251).
+
+</TabItem>
+
+</Tabs>
+
+## Bookie isolation
+
+A namespace can be isolated into user-defined groups of bookies, which
guarantees all the data that belongs to the namespace is stored in desired
bookies. The bookie affinity group uses the BookKeeper [rack-aware placement
policy](https://bookkeeper.apache.org/docs/latest/api/javadoc/org/apache/bookkeeper/client/EnsemblePlacementPolicy.html)
and it is a way to feed rack information which is stored as JSON format in
znode.
+
+You can set a bookie affinity group using one of the following methods.
+
+<Tabs
+ defaultValue="Admin CLI"
+ values={[
+ {
+ "label": "Admin CLI",
+ "value": "Admin CLI"
+ },
+ {
+ "label": "REST API",
+ "value": "REST API"
+ },
+ {
+ "label": "Java admin API",
+ "value": "Java admin API"
+ }
+]}>
+
+<TabItem value="Admin CLI">
+
+```
+
+pulsar-admin namespaces set-bookie-affinity-group options
+
+```
+
+For more information about the command `pulsar-admin namespaces
set-bookie-affinity-group options`, see
[here](https://pulsar.apache.org/tools/pulsar-admin/).
+
+**Example**
+
+```shell
+
+bin/pulsar-admin namespaces set-bookie-affinity-group public/default \
+--primary-group group-bookie1
+
+```
+
+</TabItem>
+<TabItem value="REST API">
+
+[POST
/admin/v2/namespaces/{tenant}/{namespace}/persistence/bookieAffinity](https://pulsar.apache.org/admin-rest-api/?version=2.7.0&apiversion=v2#operation/setBookieAffinityGroup)
+
+</TabItem>
+<TabItem value="Java admin API">
+
+For how to set bookie affinity group for a namespace using Java admin API, see
[here](https://github.com/apache/pulsar/blob/master/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/NamespacesImpl.java#L1164).
+
+</TabItem>
+
+</Tabs>
diff --git
a/site2/website-next/versioned_docs/version-2.7.1/administration-load-balance.md
b/site2/website-next/versioned_docs/version-2.7.1/administration-load-balance.md
new file mode 100644
index 0000000..a14142b
--- /dev/null
+++
b/site2/website-next/versioned_docs/version-2.7.1/administration-load-balance.md
@@ -0,0 +1,204 @@
+---
+id: administration-load-balance
+title: Pulsar load balance
+sidebar_label: "Load balance"
+original_id: administration-load-balance
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+## Load balance across Pulsar brokers
+
+Pulsar is an horizontally scalable messaging system, so the traffic
+in a logical cluster must be spread across all the available Pulsar brokers as
evenly as possible, which is a core requirement.
+
+You can use multiple settings and tools to control the traffic distribution
which require a bit of context to understand how the traffic is managed in
Pulsar. Though, in most cases, the core requirement mentioned above is true out
of the box and you should not worry about it.
+
+## Pulsar load manager architecture
+
+The following part introduces the basic architecture of the Pulsar load
manager.
+
+### Assign topics to brokers dynamically
+
+Topics are dynamically assigned to brokers based on the load conditions of all
brokers in the cluster.
+
+When a client starts using new topics that are not assigned to any broker, a
process is triggered to choose the best suited broker to acquire ownership of
these topics according to the load conditions.
+
+In case of partitioned topics, different partitions are assigned to different
brokers. Here "topic" means either a non-partitioned topic or one partition of
a topic.
+
+The assignment is "dynamic" because the assignment changes quickly. For
example, if the broker owning the topic crashes, the topic is reassigned
immediately to another broker. Another scenario is that the broker owning the
topic becomes overloaded. In this case, the topic is reassigned to a less
loaded broker.
+
+The stateless nature of brokers makes the dynamic assignment possible, so you
can quickly expand or shrink the cluster based on usage.
+
+#### Assignment granularity
+
+The assignment of topics or partitions to brokers is not done at the topics or
partitions level, but done at the Bundle level (a higher level). The reason is
to amortize the amount of information that you need to keep track. Based on
CPU, memory, traffic load and other indexes, topics are assigned to a
particular broker dynamically.
+
+Instead of individual topic or partition assignment, each broker takes
ownership of a subset of the topics for a namespace. This subset is called a
"*bundle*" and effectively this subset is a sharding mechanism.
+
+The namespace is the "administrative" unit: many config knobs or operations
are done at the namespace level.
+
+For assignment, a namespaces is sharded into a list of "bundles", with each
bundle comprising
+a portion of overall hash range of the namespace.
+
+Topics are assigned to a particular bundle by taking the hash of the topic
name and checking in which
+bundle the hash falls into.
+
+Each bundle is independent of the others and thus is independently assigned to
different brokers.
+
+### Create namespaces and bundles
+
+When you create a new namespace, the new namespace sets to use the default
number of bundles. You can set this in `conf/broker.conf`:
+
+```properties
+
+# When a namespace is created without specifying the number of bundle, this
+# value will be used as the default
+defaultNumberOfNamespaceBundles=4
+
+```
+
+You can either change the system default, or override it when you create a new
namespace:
+
+```shell
+
+$ bin/pulsar-admin namespaces create my-tenant/my-namespace --clusters us-west
--bundles 16
+
+```
+
+With this command, you create a namespace with 16 initial bundles. Therefore
the topics for this namespaces can immediately be spread across up to 16
brokers.
+
+In general, if you know the expected traffic and number of topics in advance,
you had better start with a reasonable number of bundles instead of waiting for
the system to auto-correct the distribution.
+
+On the same note, it is beneficial to start with more bundles than the number
of brokers, because of the hashing nature of the distribution of topics into
bundles. For example, for a namespace with 1000 topics, using something like 64
bundles achieves a good distribution of traffic across 16 brokers.
+
+### Unload topics and bundles
+
+You can "unload" a topic in Pulsar with admin operation. Unloading means to
close the topics,
+release ownership and reassign the topics to a new broker, based on current
load.
+
+When unloading happens, the client experiences a small latency blip, typically
in the order of tens of milliseconds, while the topic is reassigned.
+
+Unloading is the mechanism that the load-manager uses to perform the load
shedding, but you can also trigger the unloading manually, for example to
correct the assignments and redistribute traffic even before having any broker
overloaded.
+
+Unloading a topic has no effect on the assignment, but just closes and reopens
the particular topic:
+
+```shell
+
+pulsar-admin topics unload persistent://tenant/namespace/topic
+
+```
+
+To unload all topics for a namespace and trigger reassignments:
+
+```shell
+
+pulsar-admin namespaces unload tenant/namespace
+
+```
+
+### Split namespace bundles
+
+Since the load for the topics in a bundle might change over time, or
predicting upfront might just be hard, brokers can split bundles into two. The
new smaller bundles can be reassigned to different brokers.
+
+The splitting happens based on some tunable thresholds. Any existing bundle
that exceeds any of the threshold is a candidate to be split. By default the
newly split bundles are also immediately offloaded to other brokers, to
facilitate the traffic distribution.
+
+```properties
+
+# enable/disable namespace bundle auto split
+loadBalancerAutoBundleSplitEnabled=true
+
+# enable/disable automatic unloading of split bundles
+loadBalancerAutoUnloadSplitBundlesEnabled=true
+
+# maximum topics in a bundle, otherwise bundle split will be triggered
+loadBalancerNamespaceBundleMaxTopics=1000
+
+# maximum sessions (producers + consumers) in a bundle, otherwise bundle split
will be triggered
+loadBalancerNamespaceBundleMaxSessions=1000
+
+# maximum msgRate (in + out) in a bundle, otherwise bundle split will be
triggered
+loadBalancerNamespaceBundleMaxMsgRate=30000
+
+# maximum bandwidth (in + out) in a bundle, otherwise bundle split will be
triggered
+loadBalancerNamespaceBundleMaxBandwidthMbytes=100
+
+# maximum number of bundles in a namespace (for auto-split)
+loadBalancerNamespaceMaximumBundles=128
+
+```
+
+### Shed load automatically
+
+The support for automatic load shedding is available in the load manager of
Pulsar. This means that whenever the system recognizes a particular broker is
overloaded, the system forces some traffic to be reassigned to less loaded
brokers.
+
+When a broker is identified as overloaded, the broker forces to "unload" a
subset of the bundles, the
+ones with higher traffic, that make up for the overload percentage.
+
+For example, the default threshold is 85% and if a broker is over quota at 95%
CPU usage, then the broker unloads the percent difference plus a 5% margin:
`(95% - 85%) + 5% = 15%`.
+
+Given the selection of bundles to offload is based on traffic (as a proxy
measure for cpu, network
+and memory), broker unloads bundles for at least 15% of traffic.
+
+The automatic load shedding is enabled by default and you can disable the
automatic load shedding with this setting:
+
+```properties
+
+# Enable/disable automatic bundle unloading for load-shedding
+loadBalancerSheddingEnabled=true
+
+```
+
+Additional settings that apply to shedding:
+
+```properties
+
+# Load shedding interval. Broker periodically checks whether some traffic
should be offload from
+# some over-loaded broker to other under-loaded brokers
+loadBalancerSheddingIntervalMinutes=1
+
+# Prevent the same topics to be shed and moved to other brokers more that once
within this timeframe
+loadBalancerSheddingGracePeriodMinutes=30
+
+```
+
+#### Broker overload thresholds
+
+The determinations of when a broker is overloaded is based on threshold of
CPU, network and memory usage. Whenever either of those metrics reaches the
threshold, the system triggers the shedding (if enabled).
+
+By default, overload threshold is set at 85%:
+
+```properties
+
+# Usage threshold to determine a broker as over-loaded
+loadBalancerBrokerOverloadedThresholdPercentage=85
+
+```
+
+Pulsar gathers the usage stats from the system metrics.
+
+In case of network utilization, in some cases the network interface speed that
Linux reports is
+not correct and needs to be manually overridden. This is the case in AWS EC2
instances with 1Gbps
+NIC speed for which the OS reports 10Gbps speed.
+
+Because of the incorrect max speed, the Pulsar load manager might think the
broker has not reached the NIC capacity, while in fact the broker already uses
all the bandwidth and the traffic is slowed down.
+
+You can use the following setting to correct the max NIC speed:
+
+```properties
+
+# Override the auto-detection of the network interfaces max speed.
+# This option is useful in some environments (eg: EC2 VMs) where the max speed
+# reported by Linux is not reflecting the real bandwidth available to the
broker.
+# Since the network usage is employed by the load manager to decide when a
broker
+# is overloaded, it is important to make sure the info is correct or override
it
+# with the right value here. The configured value can be a double (eg: 0.8)
and that
+# can be used to trigger load-shedding even before hitting on NIC limits.
+loadBalancerOverrideBrokerNicSpeedGbps=
+
+```
+
+When the value is empty, Pulsar uses the value that the OS reports.
+
diff --git
a/site2/website-next/versioned_docs/version-2.7.1/administration-proxy.md
b/site2/website-next/versioned_docs/version-2.7.1/administration-proxy.md
new file mode 100644
index 0000000..821aa4d
--- /dev/null
+++ b/site2/website-next/versioned_docs/version-2.7.1/administration-proxy.md
@@ -0,0 +1,90 @@
+---
+id: administration-proxy
+title: Pulsar proxy
+sidebar_label: "Pulsar proxy"
+original_id: administration-proxy
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+Pulsar proxy is an optional gateway. Pulsar proxy is used when direction
connections between clients and Pulsar brokers are either infeasible or
undesirable. For example, when you run Pulsar in a cloud environment or on
[Kubernetes](https://kubernetes.io) or an analogous platform, you can run
Pulsar proxy.
+
+## Configure the proxy
+
+Before using the proxy, you need to configure it with the brokers addresses in
the cluster. You can configure the proxy to connect directly to service
discovery, or specify a broker URL in the configuration.
+
+### Use service discovery
+
+Pulsar uses [ZooKeeper](https://zookeeper.apache.org) for service discovery.
To connect the proxy to ZooKeeper, specify the following in `conf/proxy.conf`.
+
+```properties
+
+zookeeperServers=zk-0,zk-1,zk-2
+configurationStoreServers=zk-0:2184,zk-remote:2184
+
+```
+
+> To use service discovery, you need to open the network ACLs, so the proxy
can connects to the ZooKeeper nodes through the ZooKeeper client port (port
`2181`) and the configuration store client port (port `2184`).
+
+> However, it is not secure to use service discovery. Because if the network
ACL is open, when someone compromises a proxy, they have full access to
ZooKeeper.
+
+### Use broker URLs
+
+It is more secure to specify a URL to connect to the brokers.
+
+Proxy authorization requires access to ZooKeeper, so if you use these broker
URLs to connect to the brokers, you need to disable authorization at the Proxy
level. Brokers still authorize requests after the proxy forwards them.
+
+You can configure the broker URLs in `conf/proxy.conf` as follows.
+
+```properties
+
+brokerServiceURL=pulsar://brokers.example.com:6650
+brokerWebServiceURL=http://brokers.example.com:8080
+functionWorkerWebServiceURL=http://function-workers.example.com:8080
+
+```
+
+If you use TLS, configure the broker URLs in the following way:
+
+```properties
+
+brokerServiceURLTLS=pulsar+ssl://brokers.example.com:6651
+brokerWebServiceURLTLS=https://brokers.example.com:8443
+functionWorkerWebServiceURL=https://function-workers.example.com:8443
+
+```
+
+The hostname in the URLs provided should be a DNS entry which points to
multiple brokers or a virtual IP address, which is backed by multiple broker IP
addresses, so that the proxy does not lose connectivity to Pulsar cluster if a
single broker becomes unavailable.
+
+The ports to connect to the brokers (6650 and 8080, or in the case of TLS,
6651 and 8443) should be open in the network ACLs.
+
+Note that if you do not use functions, you do not need to configure
`functionWorkerWebServiceURL`.
+
+## Start the proxy
+
+To start the proxy:
+
+```bash
+
+$ cd /path/to/pulsar/directory
+$ bin/pulsar proxy
+
+```
+
+> You can run multiple instances of the Pulsar proxy in a cluster.
+
+## Stop the proxy
+
+Pulsar proxy runs in the foreground by default. To stop the proxy, simply stop
the process in which the proxy is running.
+
+## Proxy frontends
+
+You can run Pulsar proxy behind some kind of load-distributing frontend, such
as an
[HAProxy](https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts)
load balancer.
+
+## Use Pulsar clients with the proxy
+
+Once your Pulsar proxy is up and running, preferably behind a
load-distributing [frontend](#proxy-frontends), clients can connect to the
proxy via whichever address that the frontend uses. If the address is the DNS
address `pulsar.cluster.default`, for example, the connection URL for clients
is `pulsar://pulsar.cluster.default:6650`.
+
+For more information on Proxy configuration, refer to [Pulsar
proxy](reference-configuration.md#pulsar-proxy).
diff --git
a/site2/website-next/versioned_docs/version-2.7.1/administration-pulsar-manager.md
b/site2/website-next/versioned_docs/version-2.7.1/administration-pulsar-manager.md
new file mode 100644
index 0000000..6c8945e
--- /dev/null
+++
b/site2/website-next/versioned_docs/version-2.7.1/administration-pulsar-manager.md
@@ -0,0 +1,209 @@
+---
+id: administration-pulsar-manager
+title: Pulsar Manager
+sidebar_label: "Pulsar Manager"
+original_id: administration-pulsar-manager
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+Pulsar Manager is a web-based GUI management and monitoring tool that helps
administrators and users manage and monitor tenants, namespaces, topics,
subscriptions, brokers, clusters, and so on, and supports dynamic configuration
of multiple environments.
+
+:::note
+
+If you monitor your current stats with [Pulsar
dashboard](administration-dashboard), you can try to use Pulsar Manager
instead. Pulsar dashboard is deprecated.
+
+:::
+
+## Install
+
+The easiest way to use the Pulsar Manager is to run it inside a
[Docker](https://www.docker.com/products/docker) container.
+
+```shell
+
+docker pull apachepulsar/pulsar-manager:v0.2.0
+docker run -it \
+ -p 9527:9527 -p 7750:7750 \
+ -e
SPRING_CONFIGURATION_FILE=/pulsar-manager/pulsar-manager/application.properties
\
+ apachepulsar/pulsar-manager:v0.2.0
+
+```
+
+* `SPRING_CONFIGURATION_FILE`: Default configuration file for spring.
+
+### Set administrator account and password
+
+ ```shell
+
+ CSRF_TOKEN=$(curl http://localhost:7750/pulsar-manager/csrf-token)
+ curl \
+ -H 'X-XSRF-TOKEN: $CSRF_TOKEN' \
+ -H 'Cookie: XSRF-TOKEN=$CSRF_TOKEN;' \
+ -H "Content-Type: application/json" \
+ -X PUT http://localhost:7750/pulsar-manager/users/superuser \
+ -d '{"name": "admin", "password": "apachepulsar", "description": "test",
"email": "[email protected]"}'
+
+ ```
+
+You can find the docker image in the [Docker
Hub](https://github.com/apache/pulsar-manager/tree/master/docker) directory and
build an image from the source code as well:
+
+```
+
+git clone https://github.com/apache/pulsar-manager
+cd pulsar-manager/front-end
+npm install --save
+npm run build:prod
+cd ..
+./gradlew build -x test
+cd ..
+docker build -f docker/Dockerfile --build-arg BUILD_DATE=`date -u
+"%Y-%m-%dT%H:%M:%SZ"` --build-arg VCS_REF=`latest` --build-arg
VERSION=`latest` -t apachepulsar/pulsar-manager .
+
+```
+
+### Use custom databases
+
+If you have a large amount of data, you can use a custom database. The
following is an example of PostgreSQL.
+
+1. Initialize database and table structures using the
[file](https://github.com/apache/pulsar-manager/tree/master/src/main/resources/META-INF/sql/postgresql-schema.sql).
+
+2. Modify the [configuration
file](https://github.com/apache/pulsar-manager/blob/master/src/main/resources/application.properties)
and add PostgreSQL configuration.
+
+```
+
+spring.datasource.driver-class-name=org.postgresql.Driver
+spring.datasource.url=jdbc:postgresql://127.0.0.1:5432/pulsar_manager
+spring.datasource.username=postgres
+spring.datasource.password=postgres
+
+```
+
+3. Compile to generate a new executable jar package.
+
+```
+
+./gradlew build -x test
+
+```
+
+### Enable JWT authentication
+
+If you want to turn on JWT authentication, configure the following parameters:
+
+* `backend.jwt.token`: token for the superuser. You need to configure this
parameter during cluster initialization.
+* `jwt.broker.token.mode`: multiple modes of generating token, including
PUBLIC, PRIVATE, and SECRET.
+* `jwt.broker.public.key`: configure this option if you use the PUBLIC mode.
+* `jwt.broker.private.key`: configure this option if you use the PRIVATE mode.
+* `jwt.broker.secret.key`: configure this option if you use the SECRET mode.
+
+For more information, see [Token Authentication Admin of
Pulsar](http://pulsar.apache.org/docs/en/security-token-admin/).
+
+
+If you want to enable JWT authentication, use one of the following methods.
+
+
+* Method 1: use command-line tool
+
+```
+
+wget
https://dist.apache.org/repos/dist/release/pulsar/pulsar-manager/apache-pulsar-manager-0.2.0/apache-pulsar-manager-0.2.0-bin.tar.gz
+tar -zxvf apache-pulsar-manager-0.2.0-bin.tar.gz
+cd pulsar-manager
+tar -zxvf pulsar-manager.tar
+cd pulsar-manager
+cp -r ../dist ui
+./bin/pulsar-manager --redirect.host=http://localhost --redirect.port=9527
insert.stats.interval=600000 --backend.jwt.token=token
--jwt.broker.token.mode=PRIVATE
--jwt.broker.private.key=file:///path/broker-private.key
--jwt.broker.public.key=file:///path/broker-public.key
+
+```
+
+Firstly, [set the administrator account and
password](#set-administrator-account-and-password)
+
+Secondly, log in to Pulsar manager through http://localhost:7750/ui/index.html.
+
+* Method 2: configure the application.properties file
+
+```
+
+backend.jwt.token=token
+
+jwt.broker.token.mode=PRIVATE
+jwt.broker.public.key=file:///path/broker-public.key
+jwt.broker.private.key=file:///path/broker-private.key
+
+or
+jwt.broker.token.mode=SECRET
+jwt.broker.secret.key=file:///path/broker-secret.key
+
+```
+
+* Method 3: use Docker and enable token authentication.
+
+```
+
+export JWT_TOKEN="your-token"
+docker run -it -p 9527:9527 -p 7750:7750 -e REDIRECT_HOST=http://localhost -e
REDIRECT_PORT=9527 -e DRIVER_CLASS_NAME=org.postgresql.Driver -e
URL='jdbc:postgresql://127.0.0.1:5432/pulsar_manager' -e USERNAME=pulsar -e
PASSWORD=pulsar -e LOG_LEVEL=DEBUG -e JWT_TOKEN=$JWT_TOKEN -v $PWD:/data
apachepulsar/pulsar-manager:v0.2.0 /bin/sh
+
+```
+
+* `JWT_TOKEN`: the token of superuser configured for the broker. It is
generated by the `bin/pulsar tokens create --secret-key` or `bin/pulsar tokens
create --private-key` command.
+* `REDIRECT_HOST`: the IP address of the front-end server.
+* `REDIRECT_PORT`: the port of the front-end server.
+* `DRIVER_CLASS_NAME`: the driver class name of the PostgreSQL database.
+* `URL`: the JDBC URL of your PostgreSQL database, such as
jdbc:postgresql://127.0.0.1:5432/pulsar_manager. The docker image automatically
start a local instance of the PostgreSQL database.
+* `USERNAME`: the username of PostgreSQL.
+* `PASSWORD`: the password of PostgreSQL.
+* `LOG_LEVEL`: the level of log.
+
+* Method 4: use Docker and turn on **token authentication** and **token
management** by private key and public key.
+
+```
+
+export JWT_TOKEN="your-token"
+export PRIVATE_KEY="file:///pulsar-manager/secret/my-private.key"
+export PUBLIC_KEY="file:///pulsar-manager/secret/my-public.key"
+docker run -it -p 9527:9527 -p 7750:7750 -e REDIRECT_HOST=http://localhost -e
REDIRECT_PORT=9527 -e DRIVER_CLASS_NAME=org.postgresql.Driver -e
URL='jdbc:postgresql://127.0.0.1:5432/pulsar_manager' -e USERNAME=pulsar -e
PASSWORD=pulsar -e LOG_LEVEL=DEBUG -e JWT_TOKEN=$JWT_TOKEN -e
PRIVATE_KEY=$PRIVATE_KEY -e PUBLIC_KEY=$PUBLIC_KEY -v $PWD:/data -v
$PWD/secret:/pulsar-manager/secret apachepulsar/pulsar-manager:v0.2.0 /bin/sh
+
+```
+
+* `JWT_TOKEN`: the token of superuser configured for the broker. It is
generated by the `bin/pulsar tokens create --private-key` command.
+* `PRIVATE_KEY`: private key path mounted in container, generated by
`bin/pulsar tokens create-key-pair` command.
+* `PUBLIC_KEY`: public key path mounted in container, generated by `bin/pulsar
tokens create-key-pair` command.
+* `$PWD/secret`: the folder where the private key and public key generated by
the `bin/pulsar tokens create-key-pair` command are placed locally
+* `REDIRECT_HOST`: the IP address of the front-end server.
+* `REDIRECT_PORT`: the port of the front-end server.
+* `DRIVER_CLASS_NAME`: the driver class name of the PostgreSQL database.
+* `URL`: the JDBC URL of your PostgreSQL database, such as
jdbc:postgresql://127.0.0.1:5432/pulsar_manager. The docker image automatically
start a local instance of the PostgreSQL database.
+* `USERNAME`: the username of PostgreSQL.
+* `PASSWORD`: the password of PostgreSQL.
+* `LOG_LEVEL`: the level of log.
+
+* Method 5: use Docker and turn on **token authentication** and **token
management** by secret key.
+
+```
+
+export JWT_TOKEN="your-token"
+export SECRET_KEY="file:///pulsar-manager/secret/my-secret.key"
+docker run -it -p 9527:9527 -p 7750:7750 -e REDIRECT_HOST=http://localhost -e
REDIRECT_PORT=9527 -e DRIVER_CLASS_NAME=org.postgresql.Driver -e
URL='jdbc:postgresql://127.0.0.1:5432/pulsar_manager' -e USERNAME=pulsar -e
PASSWORD=pulsar -e LOG_LEVEL=DEBUG -e JWT_TOKEN=$JWT_TOKEN -e
SECRET_KEY=$SECRET_KEY -v $PWD:/data -v $PWD/secret:/pulsar-manager/secret
apachepulsar/pulsar-manager:v0.2.0 /bin/sh
+
+```
+
+* `JWT_TOKEN`: the token of superuser configured for the broker. It is
generated by the `bin/pulsar tokens create --secret-key` command.
+* `SECRET_KEY`: secret key path mounted in container, generated by `bin/pulsar
tokens create-secret-key` command.
+* `$PWD/secret`: the folder where the secret key generated by the `bin/pulsar
tokens create-secret-key` command are placed locally
+* `REDIRECT_HOST`: the IP address of the front-end server.
+* `REDIRECT_PORT`: the port of the front-end server.
+* `DRIVER_CLASS_NAME`: the driver class name of the PostgreSQL database.
+* `URL`: the JDBC URL of your PostgreSQL database, such as
jdbc:postgresql://127.0.0.1:5432/pulsar_manager. The docker image automatically
start a local instance of the PostgreSQL database.
+* `USERNAME`: the username of PostgreSQL.
+* `PASSWORD`: the password of PostgreSQL.
+* `LOG_LEVEL`: the level of log.
+
+* For more information about backend configurations, see
[here](https://github.com/apache/pulsar-manager/blob/master/src/README).
+* For more information about frontend configurations, see
[here](https://github.com/apache/pulsar-manager/tree/master/front-end).
+
+## Log in
+
+[Set the administrator account and
password](#set-administrator-account-and-password).
+
+Visit http://localhost:9527 to log in.
diff --git
a/site2/website-next/versioned_docs/version-2.7.1/administration-stats.md
b/site2/website-next/versioned_docs/version-2.7.1/administration-stats.md
new file mode 100644
index 0000000..029ebf2
--- /dev/null
+++ b/site2/website-next/versioned_docs/version-2.7.1/administration-stats.md
@@ -0,0 +1,68 @@
+---
+id: administration-stats
+title: Pulsar stats
+sidebar_label: "Pulsar statistics"
+original_id: administration-stats
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+## Partitioned topics
+
+|Stat|Description|
+|---|---|
+|msgRateIn| The sum of publish rates of all local and replication publishers
in messages per second.|
+|msgThroughputIn| Same as msgRateIn but in bytes per second instead of
messages per second.|
+|msgRateOut| The sum of dispatch rates of all local and replication consumers
in messages per second.|
+|msgThroughputOut| Same as msgRateOut but in bytes per second instead of
messages per second.|
+|averageMsgSize| Average message size, in bytes, from this publisher within
the last interval.|
+|storageSize| The sum of storage size of the ledgers for this topic.|
+|publishers| The list of all local publishers into the topic. Publishers can
be anywhere from zero to thousands.|
+|producerId| Internal identifier for this producer on this topic.|
+|producerName| Internal identifier for this producer, generated by the client
library.|
+|address| IP address and source port for the connection of this producer.|
+|connectedSince| Timestamp this producer is created or last reconnected.|
+|subscriptions| The list of all local subscriptions to the topic.|
+|my-subscription| The name of this subscription (client defined).|
+|msgBacklog| The count of messages in backlog for this subscription.|
+|type| This subscription type.|
+|msgRateExpired| The rate at which messages are discarded instead of
dispatched from this subscription due to TTL.|
+|consumers| The list of connected consumers for this subscription.|
+|consumerName| Internal identifier for this consumer, generated by the client
library.|
+|availablePermits| The number of messages this consumer has space for in the
listen queue of client library. A value of 0 means the queue of client library
is full and receive() is not being called. A nonzero value means this consumer
is ready to be dispatched messages.|
+|replication| This section gives the stats for cross-colo replication of this
topic.|
+|replicationBacklog| The outbound replication backlog in messages.|
+|connected| Whether the outbound replicator is connected.|
+|replicationDelayInSeconds| How long the oldest message has been waiting to be
sent through the connection, if connected is true.|
+|inboundConnection| The IP and port of the broker in the publisher connection
of remote cluster to this broker. |
+|inboundConnectedSince| The TCP connection being used to publish messages to
the remote cluster. If no local publishers are connected, this connection is
automatically closed after a minute.|
+
+
+## Topics
+
+|Stat|Description|
+|---|---|
+|entriesAddedCounter| Messages published since this broker loads this topic.|
+|numberOfEntries| Total number of messages being tracked.|
+|totalSize| Total storage size in bytes of all messages.|
+|currentLedgerEntries| Count of messages written to the ledger currently open
for writing.|
+|currentLedgerSize| Size in bytes of messages written to ledger currently open
for writing.|
+|lastLedgerCreatedTimestamp| Time when last ledger is created.|
+|lastLedgerCreationFailureTimestamp| Time when last ledger is failed.|
+|waitingCursorsCount| How many cursors are caught up and waiting for a new
message to be published.|
+|pendingAddEntriesCount| How many messages have (asynchronous) write requests
you are waiting on completion.|
+|lastConfirmedEntry| The ledgerid:entryid of the last message successfully
written. If the entryid is -1, then the ledger is opened or is being currently
opened but has no entries written yet.|
+|state| The state of the cursor ledger. Open means you have a cursor ledger
for saving updates of the markDeletePosition.|
+|ledgers| The ordered list of all ledgers for this topic holding its messages.|
+|cursors| The list of all cursors on this topic. Every subscription you saw in
the topic stats has one.|
+|markDeletePosition| The ack position: the last message the subscriber
acknowledges receiving.|
+|readPosition| The latest position of subscriber for reading message.|
+|waitingReadOp| This is true when the subscription reads the latest message
that is published to the topic and waits on new messages to be published.|
+|pendingReadOps| The counter for how many outstanding read requests to the
BookKeepers you have in progress.|
+|messagesConsumedCounter| Number of messages this cursor acks since this
broker loads this topic.|
+|cursorLedger| The ledger used to persistently store the current
markDeletePosition.|
+|cursorLedgerLastEntry| The last entryid used to persistently store the
current markDeletePosition.|
+|individuallyDeletedMessages| If Acks are done out of order, shows the ranges
of messages Acked between the markDeletePosition and the read-position.|
+|lastLedgerSwitchTimestamp| The last time the cursor ledger is rolled over.|
diff --git
a/site2/website-next/versioned_docs/version-2.7.1/administration-upgrade.md
b/site2/website-next/versioned_docs/version-2.7.1/administration-upgrade.md
new file mode 100644
index 0000000..aefdd81
--- /dev/null
+++ b/site2/website-next/versioned_docs/version-2.7.1/administration-upgrade.md
@@ -0,0 +1,172 @@
+---
+id: administration-upgrade
+title: Upgrade Guide
+sidebar_label: "Upgrade"
+original_id: administration-upgrade
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+## Upgrade guidelines
+
+Apache Pulsar is comprised of multiple components, ZooKeeper, bookies, and
brokers. These components are either stateful or stateless. You do not have to
upgrade ZooKeeper nodes unless you have special requirement. While you upgrade,
you need to pay attention to bookies (stateful), brokers and proxies
(stateless).
+
+The following are some guidelines on upgrading a Pulsar cluster. Read the
guidelines before upgrading.
+
+- Backup all your configuration files before upgrading.
+- Read guide entirely, make a plan, and then execute the plan. When you make
upgrade plan, you need to take your specific requirements and environment into
consideration.
+- Pay attention to the upgrading order of components. In general, you do not
need to upgrade your ZooKeeper or configuration store cluster (the global
ZooKeeper cluster). You need to upgrade bookies first, and then upgrade
brokers, proxies, and your clients.
+- If `autorecovery` is enabled, you need to disable `autorecovery` in the
upgrade process, and re-enable it after completing the process.
+- Read the release notes carefully for each release. Release notes contain
features, configuration changes that might impact your upgrade.
+- Upgrade a small subset of nodes of each type to canary test the new version
before upgrading all nodes of that type in the cluster. When you have upgraded
the canary nodes, run for a while to ensure that they work correctly.
+- Upgrade one data center to verify new version before upgrading all data
centers if your cluster runs in multi-cluster replicated mode.
+
+> Note: Currently, Apache Pulsar is compatible between versions.
+
+## Upgrade sequence
+
+To upgrade an Apache Pulsar cluster, follow the upgrade sequence.
+
+1. Upgrade ZooKeeper (optional)
+- Canary test: test an upgraded version in one or a small set of ZooKeeper
nodes.
+- Rolling upgrade: rollout the upgraded version to all ZooKeeper servers
incrementally, one at a time. Monitor your dashboard during the whole rolling
upgrade process.
+2. Upgrade bookies
+- Canary test: test an upgraded version in one or a small set of bookies.
+- Rolling upgrade:
+ - a. Disable `autorecovery` with the following command.
+
+ ```shell
+
+ bin/bookkeeper shell autorecovery -disable
+
+ ```
+
+
+ - b. Rollout the upgraded version to all bookies in the cluster after you
determine that a version is safe after canary.
+ - c. After you upgrade all bookies, re-enable `autorecovery` with the
following command.
+
+ ```shell
+
+ bin/bookkeeper shell autorecovery -enable
+
+ ```
+
+3. Upgrade brokers
+- Canary test: test an upgraded version in one or a small set of brokers.
+- Rolling upgrade: rollout the upgraded version to all brokers in the cluster
after you determine that a version is safe after canary.
+4. Upgrade proxies
+- Canary test: test an upgraded version in one or a small set of proxies.
+- Rolling upgrade: rollout the upgraded version to all proxies in the cluster
after you determine that a version is safe after canary.
+
+## Upgrade ZooKeeper (optional)
+While you upgrade ZooKeeper servers, you can do canary test first, and then
upgrade all ZooKeeper servers in the cluster.
+
+### Canary test
+
+You can test an upgraded version in one of ZooKeeper servers before upgrading
all ZooKeeper servers in your cluster.
+
+To upgrade ZooKeeper server to a new version, complete the following steps:
+
+1. Stop a ZooKeeper server.
+2. Upgrade the binary and configuration files.
+3. Start the ZooKeeper server with the new binary files.
+4. Use `pulsar zookeeper-shell` to connect to the newly upgraded ZooKeeper
server and run a few commands to verify if it works as expected.
+5. Run the ZooKeeper server for a few days, observe and make sure the
ZooKeeper cluster runs well.
+
+#### Canary rollback
+
+If issues occur during canary test, you can shut down the problematic
ZooKeeper node, revert the binary and configuration, and restart the ZooKeeper
with the reverted binary.
+
+### Upgrade all ZooKeeper servers
+
+After canary test to upgrade one ZooKeeper in your cluster, you can upgrade
all ZooKeeper servers in your cluster.
+
+You can upgrade all ZooKeeper servers one by one by following steps in canary
test.
+
+## Upgrade bookies
+
+While you upgrade bookies, you can do canary test first, and then upgrade all
bookies in the cluster.
+For more details, you can read Apache BookKeeper [Upgrade
guide](http://bookkeeper.apache.org/docs/latest/admin/upgrade).
+
+### Canary test
+
+You can test an upgraded version in one or a small set of bookies before
upgrading all bookies in your cluster.
+
+To upgrade bookie to a new version, complete the following steps:
+
+1. Stop a bookie.
+2. Upgrade the binary and configuration files.
+3. Start the bookie in `ReadOnly` mode to verify if the bookie of this new
version runs well for read workload.
+
+ ```shell
+
+ bin/pulsar bookie --readOnly
+
+ ```
+
+4. When the bookie runs successfully in `ReadOnly` mode, stop the bookie and
restart it in `Write/Read` mode.
+
+ ```shell
+
+ bin/pulsar bookie
+
+ ```
+
+5. Observe and make sure the cluster serves both write and read traffic.
+
+#### Canary rollback
+
+If issues occur during the canary test, you can shut down the problematic
bookie node. Other bookies in the cluster replaces this problematic bookie node
with autorecovery.
+
+### Upgrade all bookies
+
+After canary test to upgrade some bookies in your cluster, you can upgrade all
bookies in your cluster.
+
+Before upgrading, you have to decide whether to upgrade the whole cluster at
once, including downtime and rolling upgrade scenarios.
+
+In a rolling upgrade scenario, upgrade one bookie at a time. In a downtime
upgrade scenario, shut down the entire cluster, upgrade each bookie, and then
start the cluster.
+
+While you upgrade in both scenarios, the procedure is the same for each bookie.
+
+1. Stop the bookie.
+2. Upgrade the software (either new binary or new configuration files).
+2. Start the bookie.
+
+> **Advanced operations**
+> When you upgrade a large BookKeeper cluster in a rolling upgrade scenario,
upgrading one bookie at a time is slow. If you configure rack-aware or
region-aware placement policy, you can upgrade bookies rack by rack or region
by region, which speeds up the whole upgrade process.
+
+## Upgrade brokers and proxies
+
+The upgrade procedure for brokers and proxies is the same. Brokers and proxies
are `stateless`, so upgrading the two services is easy.
+
+### Canary test
+
+You can test an upgraded version in one or a small set of nodes before
upgrading all nodes in your cluster.
+
+To upgrade to a new version, complete the following steps:
+
+1. Stop a broker (or proxy).
+2. Upgrade the binary and configuration file.
+3. Start a broker (or proxy).
+
+#### Canary rollback
+
+If issues occur during canary test, you can shut down the problematic broker
(or proxy) node. Revert to the old version and restart the broker (or proxy).
+
+### Upgrade all brokers or proxies
+
+After canary test to upgrade some brokers or proxies in your cluster, you can
upgrade all brokers or proxies in your cluster.
+
+Before upgrading, you have to decide whether to upgrade the whole cluster at
once, including downtime and rolling upgrade scenarios.
+
+In a rolling upgrade scenario, you can upgrade one broker or one proxy at a
time if the size of the cluster is small. If your cluster is large, you can
upgrade brokers or proxies in batches. When you upgrade a batch of brokers or
proxies, make sure the remaining brokers and proxies in the cluster have enough
capacity to handle the traffic during upgrade.
+
+In a downtime upgrade scenario, shut down the entire cluster, upgrade each
broker or proxy, and then start the cluster.
+
+While you upgrade in both scenarios, the procedure is the same for each broker
or proxy.
+
+1. Stop the broker or proxy.
+2. Upgrade the software (either new binary or new configuration files).
+3. Start the broker or proxy.
diff --git
a/site2/website-next/versioned_docs/version-2.7.1/administration-zk-bk.md
b/site2/website-next/versioned_docs/version-2.7.1/administration-zk-bk.md
new file mode 100644
index 0000000..a4da422
--- /dev/null
+++ b/site2/website-next/versioned_docs/version-2.7.1/administration-zk-bk.md
@@ -0,0 +1,381 @@
+---
+id: administration-zk-bk
+title: ZooKeeper and BookKeeper administration
+sidebar_label: "ZooKeeper and BookKeeper"
+original_id: administration-zk-bk
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+Pulsar relies on two external systems for essential tasks:
+
+* [ZooKeeper](https://zookeeper.apache.org/) is responsible for a wide variety
of configuration-related and coordination-related tasks.
+* [BookKeeper](http://bookkeeper.apache.org/) is responsible for [persistent
storage](concepts-architecture-overview.md#persistent-storage) of message data.
+
+ZooKeeper and BookKeeper are both open-source
[Apache](https://www.apache.org/) projects.
+
+> Skip to the [How Pulsar uses ZooKeeper and
BookKeeper](#how-pulsar-uses-zookeeper-and-bookkeeper) section below for a more
schematic explanation of the role of these two systems in Pulsar.
+
+
+## ZooKeeper
+
+Each Pulsar instance relies on two separate ZooKeeper quorums.
+
+* [Local ZooKeeper](#deploy-local-zookeeper) operates at the cluster level and
provides cluster-specific configuration management and coordination. Each
Pulsar cluster needs to have a dedicated ZooKeeper cluster.
+* [Configuration Store](#deploy-configuration-store) operates at the instance
level and provides configuration management for the entire system (and thus
across clusters). An independent cluster of machines or the same machines that
local ZooKeeper uses can provide the configuration store quorum.
+
+### Deploy local ZooKeeper
+
+ZooKeeper manages a variety of essential coordination-related and
configuration-related tasks for Pulsar.
+
+To deploy a Pulsar instance, you need to stand up one local ZooKeeper cluster
*per Pulsar cluster*.
+
+To begin, add all ZooKeeper servers to the quorum configuration specified in
the [`conf/zookeeper.conf`](reference-configuration.md#zookeeper) file. Add a
`server.N` line for each node in the cluster to the configuration, where `N` is
the number of the ZooKeeper node. The following is an example for a three-node
cluster:
+
+```properties
+
+server.1=zk1.us-west.example.com:2888:3888
+server.2=zk2.us-west.example.com:2888:3888
+server.3=zk3.us-west.example.com:2888:3888
+
+```
+
+On each host, you need to specify the node ID in `myid` file of each node,
which is in `data/zookeeper` folder of each server by default (you can change
the file location via the
[`dataDir`](reference-configuration.md#zookeeper-dataDir) parameter).
+
+> See the [Multi-server setup
guide](https://zookeeper.apache.org/doc/r3.4.10/zookeeperAdmin.html#sc_zkMulitServerSetup)
in the ZooKeeper documentation for detailed information on `myid` and more.
+
+
+On a ZooKeeper server at `zk1.us-west.example.com`, for example, you can set
the `myid` value like this:
+
+```shell
+
+$ mkdir -p data/zookeeper
+$ echo 1 > data/zookeeper/myid
+
+```
+
+On `zk2.us-west.example.com` the command is `echo 2 > data/zookeeper/myid` and
so on.
+
+Once you add each server to the `zookeeper.conf` configuration and each server
has the appropriate `myid` entry, you can start ZooKeeper on all hosts (in the
background, using nohup) with the
[`pulsar-daemon`](reference-cli-tools.md#pulsar-daemon) CLI tool:
+
+```shell
+
+$ bin/pulsar-daemon start zookeeper
+
+```
+
+### Deploy configuration store
+
+The ZooKeeper cluster configured and started up in the section above is a
*local* ZooKeeper cluster that you can use to manage a single Pulsar cluster.
In addition to a local cluster, however, a full Pulsar instance also requires a
configuration store for handling some instance-level configuration and
coordination tasks.
+
+If you deploy a [single-cluster](#single-cluster-pulsar-instance) instance,
you do not need a separate cluster for the configuration store. If, however,
you deploy a [multi-cluster](#multi-cluster-pulsar-instance) instance, you need
to stand up a separate ZooKeeper cluster for configuration tasks.
+
+#### Single-cluster Pulsar instance
+
+If your Pulsar instance consists of just one cluster, then you can deploy a
configuration store on the same machines as the local ZooKeeper quorum but run
on different TCP ports.
+
+To deploy a ZooKeeper configuration store in a single-cluster instance, add
the same ZooKeeper servers that the local quorum uses to the configuration file
in
[`conf/global_zookeeper.conf`](reference-configuration.md#configuration-store)
using the same method for [local ZooKeeper](#local-zookeeper), but make sure to
use a different port (2181 is the default for ZooKeeper). The following is an
example that uses port 2184 for a three-node ZooKeeper cluster:
+
+```properties
+
+clientPort=2184
+server.1=zk1.us-west.example.com:2185:2186
+server.2=zk2.us-west.example.com:2185:2186
+server.3=zk3.us-west.example.com:2185:2186
+
+```
+
+As before, create the `myid` files for each server on
`data/global-zookeeper/myid`.
+
+#### Multi-cluster Pulsar instance
+
+When you deploy a global Pulsar instance, with clusters distributed across
different geographical regions, the configuration store serves as a highly
available and strongly consistent metadata store that can tolerate failures and
partitions spanning whole regions.
+
+The key here is to make sure the ZK quorum members are spread across at least
3 regions and that other regions run as observers.
+
+Again, given the very low expected load on the configuration store servers,
you can share the same hosts used for the local ZooKeeper quorum.
+
+For example, you can assume a Pulsar instance with the following clusters
`us-west`, `us-east`, `us-central`, `eu-central`, `ap-south`. Also you can
assume, each cluster has its own local ZK servers named such as
+
+```
+
+zk[1-3].${CLUSTER}.example.com
+
+```
+
+In this scenario you want to pick the quorum participants from few clusters
and let all the others be ZK observers. For example, to form a 7 servers
quorum, you can pick 3 servers from `us-west`, 2 from `us-central` and 2 from
`us-east`.
+
+This guarantees that writes to configuration store is possible even if one of
these regions is unreachable.
+
+The ZK configuration in all the servers looks like:
+
+```properties
+
+clientPort=2184
+server.1=zk1.us-west.example.com:2185:2186
+server.2=zk2.us-west.example.com:2185:2186
+server.3=zk3.us-west.example.com:2185:2186
+server.4=zk1.us-central.example.com:2185:2186
+server.5=zk2.us-central.example.com:2185:2186
+server.6=zk3.us-central.example.com:2185:2186:observer
+server.7=zk1.us-east.example.com:2185:2186
+server.8=zk2.us-east.example.com:2185:2186
+server.9=zk3.us-east.example.com:2185:2186:observer
+server.10=zk1.eu-central.example.com:2185:2186:observer
+server.11=zk2.eu-central.example.com:2185:2186:observer
+server.12=zk3.eu-central.example.com:2185:2186:observer
+server.13=zk1.ap-south.example.com:2185:2186:observer
+server.14=zk2.ap-south.example.com:2185:2186:observer
+server.15=zk3.ap-south.example.com:2185:2186:observer
+
+```
+
+Additionally, ZK observers need to have:
+
+```properties
+
+peerType=observer
+
+```
+
+##### Start the service
+
+Once your configuration store configuration is in place, you can start up the
service using [`pulsar-daemon`](reference-cli-tools.md#pulsar-daemon)
+
+```shell
+
+$ bin/pulsar-daemon start configuration-store
+
+```
+
+### ZooKeeper configuration
+
+In Pulsar, ZooKeeper configuration is handled by two separate configuration
files in the `conf` directory of your Pulsar installation:
`conf/zookeeper.conf` for [local ZooKeeper](#local-zookeeper) and
`conf/global-zookeeper.conf` for [configuration store](#configuration-store).
+
+#### Local ZooKeeper
+
+The [`conf/zookeeper.conf`](reference-configuration.md#zookeeper) file handles
the configuration for local ZooKeeper. The table below shows the available
parameters:
+
+|Name|Description|Default|
+|---|---|---|
+|tickTime| The tick is the basic unit of time in ZooKeeper, measured in
milliseconds and used to regulate things like heartbeats and timeouts. tickTime
is the length of a single tick. |2000|
+|initLimit| The maximum time, in ticks, that the leader ZooKeeper server
allows follower ZooKeeper servers to successfully connect and sync. The tick
time is set in milliseconds using the tickTime parameter. |10|
+|syncLimit| The maximum time, in ticks, that a follower ZooKeeper server is
allowed to sync with other ZooKeeper servers. The tick time is set in
milliseconds using the tickTime parameter. |5|
+|dataDir| The location where ZooKeeper stores in-memory database snapshots as
well as the transaction log of updates to the database. |data/zookeeper|
+|clientPort| The port on which the ZooKeeper server listens for connections.
|2181|
+|autopurge.snapRetainCount| In ZooKeeper, auto purge determines how many
recent snapshots of the database stored in dataDir to retain within the time
interval specified by autopurge.purgeInterval (while deleting the rest). |3|
+|autopurge.purgeInterval| The time interval, in hours, which triggers the
ZooKeeper database purge task. Setting to a non-zero number enables auto purge;
setting to 0 disables. Read this guide before enabling auto purge. |1|
+|maxClientCnxns| The maximum number of client connections. Increase this if
you need to handle more ZooKeeper clients. |60|
+
+
+#### Configuration Store
+
+The
[`conf/global-zookeeper.conf`](reference-configuration.md#configuration-store)
file handles the configuration for configuration store. The table below shows
the available parameters:
+
+
+## BookKeeper
+
+BookKeeper stores all durable message in Pulsar. BookKeeper is a distributed
[write-ahead log](https://en.wikipedia.org/wiki/Write-ahead_logging) WAL system
that guarantees read consistency of independent message logs calls ledgers.
Individual BookKeeper servers are also called *bookies*.
+
+> To manage message persistence, retention, and expiry in Pulsar, refer to
[cookbook](cookbooks-retention-expiry).
+
+### Hardware requirements
+
+Bookie hosts store message data on disk. To provide optimal performance,
ensure that the bookies have a suitable hardware configuration. The following
are two key dimensions of bookie hardware capacity:
+
+- Disk I/O capacity read/write
+- Storage capacity
+
+Message entries written to bookies are always synced to disk before returning
an acknowledgement to the Pulsar broker by default. To ensure low write
latency, BookKeeper is designed to use multiple devices:
+
+- A **journal** to ensure durability. For sequential writes, it is critical to
have fast [fsync](https://linux.die.net/man/2/fsync) operations on bookie
hosts. Typically, small and fast [solid-state
drives](https://en.wikipedia.org/wiki/Solid-state_drive) (SSDs) should suffice,
or [hard disk drives](https://en.wikipedia.org/wiki/Hard_disk_drive) (HDDs)
with a [RAID](https://en.wikipedia.org/wiki/RAID) controller and a
battery-backed write cache. Both solutions can reach fsync latency of [...]
+- A **ledger storage device** stores data. Writes happen in the background, so
write I/O is not a big concern. Reads happen sequentially most of the time and
the backlog is drained only in case of consumer drain. To store large amounts
of data, a typical configuration involves multiple HDDs with a RAID controller.
+
+### Configure BookKeeper
+
+You can configure BookKeeper bookies using the
[`conf/bookkeeper.conf`](reference-configuration.md#bookkeeper) configuration
file. When you configure each bookie, ensure that the
[`zkServers`](reference-configuration.md#bookkeeper-zkServers) parameter is set
to the connection string for local ZooKeeper of the Pulsar cluster.
+
+The minimum configuration changes required in `conf/bookkeeper.conf` are as
follows:
+
+```properties
+
+# Change to point to journal disk mount point
+journalDirectory=data/bookkeeper/journal
+
+# Point to ledger storage disk mount point
+ledgerDirectories=data/bookkeeper/ledgers
+
+# Point to local ZK quorum
+zkServers=zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181
+
+```
+
+To change the ZooKeeper root path that BookKeeper uses, use
`zkLedgersRootPath=/MY-PREFIX/ledgers` instead of
`zkServers=localhost:2181/MY-PREFIX`.
+
+> For more information about BookKeeper, refer to the official [BookKeeper
docs](http://bookkeeper.apache.org).
+
+### Deploy BookKeeper
+
+BookKeeper provides [persistent message
storage](concepts-architecture-overview.md#persistent-storage) for Pulsar. Each
Pulsar broker has its own cluster of bookies. The BookKeeper cluster shares a
local ZooKeeper quorum with the Pulsar cluster.
+
+### Start bookies manually
+
+You can start a bookie in the foreground or as a background daemon.
+
+To start a bookie in the foreground, use the
[`bookkeeper`](reference-cli-tools.md#bookkeeper) CLI tool:
+
+```bash
+
+$ bin/bookkeeper bookie
+
+```
+
+To start a bookie in the background, use the
[`pulsar-daemon`](reference-cli-tools.md#pulsar-daemon) CLI tool:
+
+```bash
+
+$ bin/pulsar-daemon start bookie
+
+```
+
+You can verify whether the bookie works properly with the `bookiesanity`
command for the [BookKeeper shell](reference-cli-tools.md#bookkeeper-shell):
+
+```shell
+
+$ bin/bookkeeper shell bookiesanity
+
+```
+
+When you use this command, you create a new ledger on the local bookie, write
a few entries, read them back and finally delete the ledger.
+
+### Decommission bookies cleanly
+
+Before you decommission a bookie, you need to check your environment and meet
the following requirements.
+
+1. Ensure the state of your cluster supports decommissioning the target
bookie. Check if `EnsembleSize >= Write Quorum >= Ack Quorum` is `true` with
one less bookie.
+
+2. Ensure the target bookie is listed after using the `listbookies` command.
+
+3. Ensure that no other process is ongoing (upgrade etc).
+
+And then you can decommission bookies safely. To decommission bookies,
complete the following steps.
+
+1. Log in to the bookie node, check if there are underreplicated ledgers. The
decommission command force to replicate the underreplicated ledgers.
+`$ bin/bookkeeper shell listunderreplicated`
+
+2. Stop the bookie by killing the bookie process. Make sure that no
liveness/readiness probes setup for the bookies to spin them back up if you
deploy it in a Kubernetes environment.
+
+3. Run the decommission command.
+ - If you have logged in to the node to be decommissioned, you do not need
to provide `-bookieid`.
+ - If you are running the decommission command for the target bookie node
from another bookie node, you should mention the target bookie ID in the
arguments for `-bookieid`
+ `$ bin/bookkeeper shell decommissionbookie`
+ or
+ `$ bin/bookkeeper shell decommissionbookie -bookieid <target bookieid>`
+
+4. Validate that no ledgers are on the decommissioned bookie.
+`$ bin/bookkeeper shell listledgers -bookieid <target bookieid>`
+
+You can run the following command to check if the bookie you have
decommissioned is listed in the bookies list:
+
+```bash
+
+./bookkeeper shell listbookies -rw -h
+./bookkeeper shell listbookies -ro -h
+
+```
+
+## BookKeeper persistence policies
+
+In Pulsar, you can set *persistence policies* at the namespace level, which
determines how BookKeeper handles persistent storage of messages. Policies
determine four things:
+
+* The number of acks (guaranteed copies) to wait for each ledger entry.
+* The number of bookies to use for a topic.
+* The number of writes to make for each ledger entry.
+* The throttling rate for mark-delete operations.
+
+### Set persistence policies
+
+You can set persistence policies for BookKeeper at the
[namespace](reference-terminology.md#namespace) level.
+
+#### Pulsar-admin
+
+Use the
[`set-persistence`](reference-pulsar-admin.md#namespaces-set-persistence)
subcommand and specify a namespace as well as any policies that you want to
apply. The available flags are:
+
+Flag | Description | Default
+:----|:------------|:-------
+`-a`, `--bookkeeper-ack-quorum` | The number of acks (guaranteed copies) to
wait on for each entry | 0
+`-e`, `--bookkeeper-ensemble` | The number of
[bookies](reference-terminology.md#bookie) to use for topics in the namespace | 0
+`-w`, `--bookkeeper-write-quorum` | The number of writes to make for each
entry | 0
+`-r`, `--ml-mark-delete-max-rate` | Throttling rate for mark-delete operations
(0 means no throttle) | 0
+
+The following is an example:
+
+```shell
+
+$ pulsar-admin namespaces set-persistence my-tenant/my-ns \
+ --bookkeeper-ack-quorum 3 \
+ --bookeeper-ensemble 2
+
+```
+
+#### REST API
+
+{@inject:
endpoint|POST|/admin/v2/namespaces/:tenant/:namespace/persistence|operation/setPersistence?version=@pulsar:version_number@}
+
+#### Java
+
+```java
+
+int bkEnsemble = 2;
+int bkQuorum = 3;
+int bkAckQuorum = 2;
+double markDeleteRate = 0.7;
+PersistencePolicies policies =
+ new PersistencePolicies(ensemble, quorum, ackQuorum, markDeleteRate);
+admin.namespaces().setPersistence(namespace, policies);
+
+```
+
+### List persistence policies
+
+You can see which persistence policy currently applies to a namespace.
+
+#### Pulsar-admin
+
+Use the
[`get-persistence`](reference-pulsar-admin.md#namespaces-get-persistence)
subcommand and specify the namespace.
+
+The following is an example:
+
+```shell
+
+$ pulsar-admin namespaces get-persistence my-tenant/my-ns
+{
+ "bookkeeperEnsemble": 1,
+ "bookkeeperWriteQuorum": 1,
+ "bookkeeperAckQuorum", 1,
+ "managedLedgerMaxMarkDeleteRate": 0
+}
+
+```
+
+#### REST API
+
+{@inject:
endpoint|GET|/admin/v2/namespaces/:tenant/:namespace/persistence|operation/getPersistence?version=@pulsar:version_number@}
+
+#### Java
+
+```java
+
+PersistencePolicies policies = admin.namespaces().getPersistence(namespace);
+
+```
+
+## How Pulsar uses ZooKeeper and BookKeeper
+
+This diagram illustrates the role of ZooKeeper and BookKeeper in a Pulsar
cluster:
+
+
+
+Each Pulsar cluster consists of one or more message brokers. Each broker
relies on an ensemble of bookies.
diff --git a/site2/website-next/versioned_sidebars/version-2.7.1-sidebars.json
b/site2/website-next/versioned_sidebars/version-2.7.1-sidebars.json
index 5244cdc..a95a781 100644
--- a/site2/website-next/versioned_sidebars/version-2.7.1-sidebars.json
+++ b/site2/website-next/versioned_sidebars/version-2.7.1-sidebars.json
@@ -295,6 +295,44 @@
"id": "version-2.7.1/deploy-monitoring"
}
]
+ },
+ {
+ "type": "category",
+ "label": "Administration",
+ "items": [
+ {
+ "type": "doc",
+ "id": "version-2.7.1/administration-zk-bk"
+ },
+ {
+ "type": "doc",
+ "id": "version-2.7.1/administration-geo"
+ },
+ {
+ "type": "doc",
+ "id": "version-2.7.1/administration-pulsar-manager"
+ },
+ {
+ "type": "doc",
+ "id": "version-2.7.1/administration-stats"
+ },
+ {
+ "type": "doc",
+ "id": "version-2.7.1/administration-load-balance"
+ },
+ {
+ "type": "doc",
+ "id": "version-2.7.1/administration-proxy"
+ },
+ {
+ "type": "doc",
+ "id": "version-2.7.1/administration-upgrade"
+ },
+ {
+ "type": "doc",
+ "id": "version-2.7.1/administration-isolation"
+ }
+ ]
}
]
}
\ No newline at end of file
