This is an automated email from the ASF dual-hosted git repository.
tison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
The following commit(s) were added to refs/heads/main by this push:
new e42133a1154 Fix script and previous content (#701)
e42133a1154 is described below
commit e42133a11543ce6ba26258d47679ac56cefc5de5
Author: tison <[email protected]>
AuthorDate: Thu Sep 7 16:02:08 2023 +0800
Fix script and previous content (#701)
Signed-off-by: tison <[email protected]>
---
static/reference/3.0.x/cli.md | 10 +-
static/reference/3.1.x/client/README.md | 28 +
static/reference/{next => 3.1.x}/config/README.md | 35 +-
.../config/reference-configuration-bookkeeper.md | 583 +++++++++++++++++++++
.../config/reference-configuration-log4j-shell.md | 37 ++
.../3.1.x/config/reference-configuration-log4j.md | 80 +++
.../config/reference-configuration-zookeeper.md | 83 +++
static/reference/3.1.x/pulsar-admin/README.md | 16 +
static/reference/3.1.x/pulsar-client/README.md | 12 +
static/reference/3.1.x/pulsar-perf/README.md | 12 +
.../reference/3.1.x/pulsar-shell/pulsar-shell.md | 83 ++-
static/reference/3.1.x/pulsar/README.md | 1 +
static/reference/next/cli.md | 10 +-
static/reference/next/client/README.md | 14 +-
static/reference/next/config/README.md | 35 +-
static/reference/next/pulsar-admin/README.md | 14 +-
static/reference/next/pulsar-client/README.md | 12 +-
static/reference/next/pulsar-perf/README.md | 12 +-
static/reference/next/pulsar-shell/pulsar-shell.md | 83 ++-
tools/pytools/bin/reference-doc-generator.py | 19 +-
20 files changed, 1013 insertions(+), 166 deletions(-)
diff --git a/static/reference/3.0.x/cli.md b/static/reference/3.0.x/cli.md
index e79763044aa..520d14e5b03 100644
--- a/static/reference/3.0.x/cli.md
+++ b/static/reference/3.0.x/cli.md
@@ -1,9 +1,3 @@
-Pulsar offers several command-line tools that help you manage Pulsar and
BookKeeper, produce and consume messages, administer Pulsar entities, test
Pulsar, and more.
+# Welcome
-|Goal|Tool
-|---|---
-Manage Pulsar| - `pulsar` <br/><br/> -
[`pulsar-daemon`](/next/pulsar-daemon/pulsar-daemon) <br/><br/> -
[`pulsar-shell`](/next/pulsar-shell/pulsar-shell)
-Produce and consume messages |
[`pulsar-client`](/next/pulsar-client/pulsar-client)
-Administer Pulsar entities | `pulsar-admin`
-Test Pulsar | [`pulsar-perf`](/next/pulsar-perf/pulsar-perf)
-Manage BookKeeper | [`bookkeper`](/next/bookkeeper/bookkeeper)
\ No newline at end of file
+> Welcome to Pulsar CLI Tools Docs.
diff --git a/static/reference/3.1.x/client/README.md
b/static/reference/3.1.x/client/README.md
new file mode 100644
index 00000000000..71df588c6f8
--- /dev/null
+++ b/static/reference/3.1.x/client/README.md
@@ -0,0 +1,28 @@
+## Client Configurations
+
+If you create a Java client, producer, consumer, or reader, you can use the
`loadConf` configurations.
+
+- [Client
configurations](/@pulsar:version_reference@/client/client-configuration-client)
+- [Producer
configurations](/@pulsar:version_reference@/client/client-configuration-producer)
+- [Consumer
configurations](/@pulsar:version_reference@/client/client-configuration-consumer)
+- [Reader
configurations](/@pulsar:version_reference@/client/client-configuration-reader)
+
+### Set client memory allocator configs using Java system properties
+
+You can set the client memory allocator configurations through Java
properties.<br />
+
+| Property | Type | <div style="width:
110pt"> Description </div>
[...]
+|-----------------------------------------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| `pulsar.allocator.pooled` | String | If set to `true`, the
client uses a direct memory pool. <br /> If set to `false`, the client uses a
heap memory without pool.
[...]
+| `pulsar.allocator.exit_on_oom` | String | Whether to exit the JVM
when OOM happens.
[...]
+| `pulsar.allocator.leak_detection` | String | The leak detection policy
for Pulsar bytebuf allocator. <li> **Disabled**: No leak detection and no
overhead. </li> <li> **Simple**: Instruments 1% of the allocated buffer to
track for leaks. </li> <li> **Advanced**: Instruments 1% of the allocated
buffer to track for leaks, reporting stack traces of places where the buffer is
used. </li> <li> **Paranoid**: Instruments 100% of the allocated buffer to
track for leaks, reporting stack tr [...]
+| `pulsar.allocator.out_of_memory_policy` | String | When an OOM occurs, the
client throws an exception or fallbacks to heap.
[...]
+
+**Example**
+
+```conf
+Dpulsar.allocator.pooled=true
+Dpulsar.allocator.exit_on_oom=false
+Dpulsar.allocator.leak_detection=Disabled
+Dpulsar.allocator.out_of_memory_policy=ThrowException
+```
diff --git a/static/reference/next/config/README.md
b/static/reference/3.1.x/config/README.md
similarity index 76%
copy from static/reference/next/config/README.md
copy to static/reference/3.1.x/config/README.md
index 807c89e69b6..02de3826186 100644
--- a/static/reference/next/config/README.md
+++ b/static/reference/3.1.x/config/README.md
@@ -1,6 +1,7 @@
## Pulsar configurations
-You can manage Pulsar configurations through configuration files in the
[`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a
Pulsar installation.
+You can manage Pulsar configurations through configuration files in
+the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a
Pulsar installation.
-
[BookKeeper](/@pulsar:version_reference@/config/reference-configuration-bookkeeper)
- [Broker](/@pulsar:version_reference@/config/reference-configuration-broker)
@@ -14,7 +15,8 @@ You can manage Pulsar configurations through configuration
files in the [`conf`]
### Override client configurations
-If you want to override the configurations of clients internal to brokers,
websockets, and proxies, you can use the following prefix.
+If you want to override the configurations of clients internal to brokers,
websockets, and proxies, you can use the
+following prefix.
| Prefix | Description
|
|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
@@ -23,21 +25,23 @@ If you want to override the configurations of clients
internal to brokers, webso
> Notes:
> * This override feature only applies to Pulsar 2.10.1 and later versions.
-> * When running the function worker within the broker, you have to configure
those clients by using the `functions_worker.yml` file. These prefixed
configurations do not apply to any of those clients.
+> * When running the function worker within the broker, you have to configure
those clients by using
+ the `functions_worker.yml` file. These prefixed configurations do not
apply to any of those clients.
### Set specific configurations using Java system property
You can set specific configurations through Java properties.
-| **Property** | **Description**
|
-| ------------------ |
------------------------------------------------------------ |
+| **Property** | **Description**
|
+|--------------------|---------------------------------------------------------------------------------------------------------------|
| pulsar.enableUring | Use `io_uring` instead of `epoll` as network IO mode.
`-Dpulsar.enableUring=1` means this feature is enabled. |
## Broker load balancing configurations
-Below is a brief summary of configurations for broker load balancing.
+Below is a brief summary of configurations for broker load balancing.
-For detailed descriptions of each configuration, see [Broker load balancing |
Configurations](pathname:///reference/#/@pulsar:version_reference@/config/reference-configuration-broker).
+For detailed descriptions of each configuration,
+see [Broker load balancing |
Configurations](pathname:///reference/#/@pulsar:version_reference@/config/reference-configuration-broker).
> Note
> Configurations with an asterisk (*) are only available in the extensible
> load balancer.
@@ -53,14 +57,13 @@ For detailed descriptions of each configuration, see
[Broker load balancing | Co
- loadBalancerMemoryResourceWeight
- loadBalancerDirectMemoryResourceWeight
- loadBalancerHistoryResourcePercentage
-
+
### TopK bundle load data
- loadBalancerReportUpdateMinIntervalMillis
- statsUpdateFrequencyInSecs
- loadBalancerMaxNumberOfBundlesInBundleLoadReport*
-
### Bundle-broker assignment
- loadBalancerAverageResourceUsageDifferenceThresholdPercentage
@@ -76,11 +79,13 @@ For detailed descriptions of each configuration, see
[Broker load balancing | Co
- loadBalancerSheddingIntervalMinutes
- loadBalancerSheddingGracePeriodMinutes
- loadBalancerBrokerOverloadedThresholdPercentage
-- loadBalancerLoadSheddingStrategy. The default value is
[org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder](https://github.com/apache/pulsar/blob/782e91fe327efe2c9c9107d6c679c2837d43935b/conf/broker.conf#L1324).
Available values are:
- - `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
- - `org.apache.pulsar.broker.loadbalance.impl.UniformLoadShedder`
- - `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder`
- -
`org.apache.pulsar.broker.loadbalance.extensions.scheduler.TransferShedder`
+- loadBalancerLoadSheddingStrategy. The default value
+ is
[org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder](https://github.com/apache/pulsar/blob/782e91fe327efe2c9c9107d6c679c2837d43935b/conf/broker.conf#L1324).
+ Available values are:
+ - `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
+ - `org.apache.pulsar.broker.loadbalance.impl.UniformLoadShedder`
+ - `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder`
+ - `org.apache.pulsar.broker.loadbalance.extensions.scheduler.TransferShedder`
- loadBalancerTransferEnabled*
- loadBalancerBrokerLoadTargetStd*
- loadBalancerSheddingConditionHitCountThreshold*
@@ -100,4 +105,4 @@ For detailed descriptions of each configuration, see
[Broker load balancing | Co
- loadBalancerNamespaceMaximumBundles
- loadBalancerSplitIntervalMinutes*
- loadBalancerNamespaceBundleSplitConditionHitCountThreshold*
-- loadBalancerMaxNumberOfBundlesToSplitPerCycle*
\ No newline at end of file
+- loadBalancerMaxNumberOfBundlesToSplitPerCycle*
diff --git
a/static/reference/3.1.x/config/reference-configuration-bookkeeper.md
b/static/reference/3.1.x/config/reference-configuration-bookkeeper.md
new file mode 100644
index 00000000000..28e85dcaf6c
--- /dev/null
+++ b/static/reference/3.1.x/config/reference-configuration-bookkeeper.md
@@ -0,0 +1,583 @@
+# BookKeeper
+
+BookKeeper is a replicated log storage system that Pulsar uses for durable
storage of all messages.
+
+### bookiePort
+
+The port on which the bookie server listens.
+
+**Default**: 3181
+
+### allowLoopback
+
+Whether the bookie is allowed to use a loopback interface as its primary
interface (that is the interface used to establish its identity). By default,
loopback interfaces are not allowed to work as the primary interface. Using a
loopback interface as the primary interface usually indicates a configuration
error. For example, it’s fairly common in some VPS setups to not configure a
hostname or to have the hostname resolve to `127.0.0.1`. If this is the case,
then all bookies in the cluste [...]
+
+**Default**: false
+
+### listeningInterface
+
+The network interface on which the bookie listens. By default, the bookie
listens on all interfaces.
+
+**Default**: eth0
+
+### advertisedAddress
+
+Configure a specific hostname or IP address that the bookie should use to
advertise itself to clients. By default, the bookie advertises either its own
IP address or hostname according to the `listeningInterface` and
`useHostNameAsBookieID` settings.
+
+**Default**: N/A
+
+### allowMultipleDirsUnderSameDiskPartition
+
+Configure the bookie to enable/disable multiple ledger/index/journal
directories in the same filesystem disk partition.
+
+**Default**: false
+
+### minUsableSizeForIndexFileCreation
+
+The minimum safe usable size available in index directory for bookie to create
index files while replaying journal at the time of bookie starts in Readonly
Mode (in bytes).
+
+**Default**: 1073741824
+
+### journalDirectory
+
+The directory where BookKeeper outputs its write-ahead log (WAL).
+
+**Default**: data/bookkeeper/journal
+
+### journalDirectories
+
+Directories that BookKeeper outputs its write ahead log. Multiple directories
are available, being separated by `,`. For example:
`journalDirectories=/tmp/bk-journal1,/tmp/bk-journal2`. If `journalDirectories`
is set, the bookies skip `journalDirectory` and use this setting directory.
+
+**Default**: /tmp/bk-journal
+
+### ledgerDirectories
+
+The directory where BookKeeper outputs ledger snapshots. This could define
multiple directories to store snapshots separated by `,`, for example
`ledgerDirectories=/tmp/bk1-data,/tmp/bk2-data`. Ideally, ledger dirs and the
journal dir are each in a different device, which reduces the contention
between random I/O and sequential write. It is possible to run with a single
disk, but performance will be significantly lower.
+
+**Default**: data/bookkeeper/ledgers
+
+### ledgerManagerType
+
+The type of ledger manager used to manage how ledgers are stored, managed, and
garbage collected. See [BookKeeper
Internals](https://bookkeeper.apache.org/docs/next/getting-started/concepts)
for more info.
+
+**Default**: hierarchical
+
+### zkLedgersRootPath
+
+The root ZooKeeper path used to store ledger metadata. This parameter is used
by the ZooKeeper-based ledger manager as a root znode to store all ledgers.
+
+**Default**: /ledgers
+
+### ledgerStorageClass
+
+Ledger storage implementation class
+
+**Default**: org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage
+
+### entryLogFilePreallocationEnabled
+
+Enable or disable entry logger preallocation
+
+**Default**: true
+
+### logSizeLimit
+
+Max file size of the entry logger, in bytes. A new entry log file will be
created when the old one reaches the file size limitation.
+
+**Default**: 1073741824
+
+### minorCompactionThreshold
+
+Threshold of minor compaction. Entry log files whose remaining size percentage
reaches below this threshold will be compacted in a minor compaction. If set to
less than zero, the minor compaction is disabled.
+
+**Default**: 0.2
+
+### minorCompactionInterval
+
+Time interval to run minor compaction, in seconds. If set to less than zero,
the minor compaction is disabled. Note: should be greater than gcWaitTime.
+
+**Default**: 3600
+
+### majorCompactionThreshold
+
+The threshold of major compaction. Entry log files whose remaining size
percentage reaches below this threshold will be compacted in a major
compaction. Those entry log files whose remaining size percentage is still
higher than the threshold will never be compacted. If set to less than zero,
the major compaction is disabled.
+
+**Default**: 0.5
+
+### majorCompactionInterval
+
+The time interval to run major compaction, in seconds. If set to less than
zero, the major compaction is disabled. Note: should be greater than gcWaitTime.
+
+**Default**: 86400
+
+### readOnlyModeEnabled
+
+If `readOnlyModeEnabled=true`, then on all full ledger disks, bookie will be
converted to read-only mode and serve only read requests. Otherwise the bookie
will be shutdown.
+
+**Default**: true
+
+### forceReadOnlyBookie
+
+Whether the bookie is force started in read only mode.
+
+**Default**: false
+
+### persistBookieStatusEnabled
+
+Persist the bookie status locally on the disks. So the bookies can keep their
status upon restarts.
+
+**Default**: false
+
+### compactionMaxOutstandingRequests
+
+Sets the maximum number of entries that can be compacted without flushing.
When compacting, the entries are written to the entrylog and the new offsets
are cached in memory. Once the entrylog is flushed the index is updated with
the new offsets. This parameter controls the number of entries added to the
entrylog before a flush is forced. A higher value for this parameter means more
memory will be used for offsets. Each offset consists of 3 longs. This
parameter should not be modified unl [...]
+
+**Default**: 100000
+
+### compactionRate
+
+The rate at which compaction will read entries, in adds per second.
+
+**Default**: 1000
+
+### isThrottleByBytes
+
+Throttle compaction by bytes or by entries.
+
+**Default**: false
+
+### compactionRateByEntries
+
+The rate at which compaction will read entries, in adds per second.
+
+**Default**: 1000
+
+### compactionRateByBytes
+
+Set the rate at which compaction reads entries. The unit is bytes added per
second.
+
+**Default**: 1000000
+
+### journalMaxSizeMB
+
+Max file size of journal file, in megabytes. A new journal file will be
created when the old one reaches the file size limitation.
+
+**Default**: 2048
+
+### journalMaxBackups
+
+The max number of old journal files to keep. Keeping a number of old journal
files would help data recovery in special cases.
+
+**Default**: 5
+
+### journalPreAllocSizeMB
+
+How space to pre-allocate at a time in the journal.
+
+**Default**: 16
+
+### journalWriteBufferSizeKB
+
+The of the write buffers used for the journal.
+
+**Default**: 64
+
+### journalRemoveFromPageCache
+
+Whether pages should be removed from the page cache after force write.
+
+**Default**: true
+
+### journalAdaptiveGroupWrites
+
+Whether to group journal force writes, which optimizes group commit for higher
throughput.
+
+**Default**: true
+
+### journalMaxGroupWaitMSec
+
+The maximum latency to impose on a journal write to achieve grouping.
+
+**Default**: 1
+
+### journalAlignmentSize
+
+All the journal writes and commits should be aligned to given size
+
+**Default**: 4096
+
+### journalBufferedWritesThreshold
+
+Maximum writes to buffer to achieve grouping
+
+**Default**: 524288
+
+### journalFlushWhenQueueEmpty
+
+If we should flush the journal when journal queue is empty
+
+**Default**: false
+
+### numJournalCallbackThreads
+
+The number of threads that should handle journal callbacks
+
+**Default**: 8
+
+### openLedgerRereplicationGracePeriod
+
+The grace period, in milliseconds, that the replication worker waits before
fencing and replicating a ledger fragment that's still being written to upon
bookie failure.
+
+**Default**: 30000
+
+### rereplicationEntryBatchSize
+
+The number of max entries to keep in fragment for re-replication
+
+**Default**: 100
+
+### autoRecoveryDaemonEnabled
+
+Whether the bookie itself can start auto-recovery service.
+
+**Default**: true
+
+### lostBookieRecoveryDelay
+
+How long to wait, in seconds, before starting auto recovery of a lost bookie.
+
+**Default**: 0
+
+### gcWaitTime
+
+How long the interval to trigger next garbage collection, in milliseconds.
Since garbage collection is running in background, too frequent gc will heart
performance. It is better to give a higher number of gc interval if there is
enough disk capacity.
+
+**Default**: 900000
+
+### gcOverreplicatedLedgerWaitTime
+
+How long the interval to trigger next garbage collection of overreplicated
ledgers, in milliseconds. This should not be run very frequently since we read
the metadata for all the ledgers on the bookie from zk.
+
+**Default**: 86400000
+
+### flushInterval
+
+How long the interval to flush ledger index pages to disk, in milliseconds.
Flushing index files will introduce much random disk I/O. If separating journal
dir and ledger dirs each on different devices, flushing would not affect
performance. But if putting journal dir and ledger dirs on same device,
performance degrade significantly on too frequent flushing. You can consider
increment flush interval to get better performance, but you need to pay more
time on bookie server restart after failure.
+
+**Default**: 60000
+
+### bookieDeathWatchInterval
+
+Interval to watch whether bookie is dead or not, in milliseconds
+
+**Default**: 1000
+
+### allowStorageExpansion
+
+Allow the bookie storage to expand. Newly added ledger and index dirs must be
empty.
+
+**Default**: false
+
+### zkServers
+
+A list of one of more servers on which zookeeper is running. The server list
can be comma separated values, for example:
zkServers=zk1:2181,zk2:2181,zk3:2181.
+
+**Default**: localhost:2181
+
+### zkTimeout
+
+ZooKeeper client session timeout in milliseconds Bookie server will exit if it
received SESSION_EXPIRED because it was partitioned off from ZooKeeper for more
than the session timeout JVM garbage collection, disk I/O will cause
SESSION_EXPIRED. Increment this value could help avoiding this issue
+
+**Default**: 30000
+
+### zkRetryBackoffStartMs
+
+The start time that the Zookeeper client backoff retries in milliseconds.
+
+**Default**: 1000
+
+### zkRetryBackoffMaxMs
+
+The maximum time that the Zookeeper client backoff retries in milliseconds.
+
+**Default**: 10000
+
+### zkEnableSecurity
+
+Set ACLs on every node written on ZooKeeper, allowing users to read and write
BookKeeper metadata stored on ZooKeeper. In order to make ACLs work you need to
setup ZooKeeper JAAS authentication. All the bookies and Client need to share
the same user, and this is usually done using Kerberos authentication. See
ZooKeeper documentation.
+
+**Default**: false
+
+### httpServerEnabled
+
+The flag enables/disables starting the admin http server.
+
+**Default**: false
+
+### httpServerPort
+
+The HTTP server port to listen on. By default, the value is `8080`. If you
want to keep it consistent with the Prometheus stats provider, you can set it
to `8000`.
+
+**Default**: 8080
+
+### httpServerClass
+
+The http server class.
+
+**Default**: org.apache.bookkeeper.http.vertx.VertxHttpServer
+
+### serverTcpNoDelay
+
+This settings is used to enabled/disabled Nagle’s algorithm, which is a means
of improving the efficiency of TCP/IP networks by reducing the number of
packets that need to be sent over the network. If you are sending many small
messages, such that more than one can fit in a single IP packet, setting
server.tcpnodelay to false to enable Nagle algorithm can provide better
performance.
+
+**Default**: true
+
+### serverSockKeepalive
+
+This setting is used to send keep-alive messages on connection-oriented
sockets.
+
+**Default**: true
+
+### serverTcpLinger
+
+The socket linger timeout on close. When enabled, a close or shutdown will not
return until all queued messages for the socket have been successfully sent or
the linger timeout has been reached. Otherwise, the call returns immediately
and the closing is done in the background.
+
+**Default**: 0
+
+### byteBufAllocatorSizeMax
+
+The maximum buf size of the received ByteBuf allocator.
+
+**Default**: 1048576
+
+### nettyMaxFrameSizeBytes
+
+The maximum netty frame size in bytes. Any message received larger than this
will be rejected.
+
+**Default**: 5253120
+
+### openFileLimit
+
+Max number of ledger index files could be opened in bookie server If number of
ledger index files reaches this limitation, bookie server started to swap some
ledgers from memory to disk. Too frequent swap will affect performance. You can
tune this number to gain performance according your requirements.
+
+**Default**: 0
+
+### pageSize
+
+Size of a index page in ledger cache, in bytes A larger index page can improve
performance writing page to disk, which is efficient when you have small number
of ledgers and these ledgers have similar number of entries. If you have large
number of ledgers and each ledger has fewer entries, smaller index page would
improve memory usage.
+
+**Default**: 8192
+
+### pageLimit
+
+How many index pages provided in ledger cache If number of index pages reaches
this limitation, bookie server starts to swap some ledgers from memory to disk.
You can increment this value when you found swap became more frequent. But make
sure pageLimit\*pageSize should not more than JVM max memory limitation,
otherwise you would got OutOfMemoryException. In general, incrementing
pageLimit, using smaller index page would gain better performance in lager
number of ledgers with fewer entri [...]
+
+**Default**: 0
+
+### readOnlyModeEnabled
+
+If all ledger directories configured are full, then support only read requests
for clients. If "readOnlyModeEnabled=true" then on all ledger disks full,
bookie will be converted to read-only mode and serve only read requests.
Otherwise the bookie will be shutdown. By default this will be disabled.
+
+**Default**: true
+
+### diskUsageThreshold
+
+For each ledger dir, maximum disk space which can be used. Default is 0.95f.
i.e. 95% of disk can be used at most after which nothing will be written to
that partition. If all ledger dir partitions are full, then bookie will turn to
readonly mode if ‘readOnlyModeEnabled=true’ is set, else it will shutdown.
Valid values should be in between 0 and 1 (exclusive).
+
+**Default**: 0.95
+
+### diskCheckInterval
+
+Disk check interval in milli seconds, interval to check the ledger dirs usage.
+
+**Default**: 10000
+
+### auditorPeriodicCheckInterval
+
+Interval at which the auditor will do a check of all ledgers in the cluster.
By default this runs once a week. The interval is set in seconds. To disable
the periodic check completely, set this to 0. Note that periodic checking will
put extra load on the cluster, so it should not be run more frequently than
once a day.
+
+**Default**: 604800
+
+### sortedLedgerStorageEnabled
+
+Whether sorted-ledger storage is enabled.
+
+**Default**: true
+
+### auditorPeriodicBookieCheckInterval
+
+The interval between auditor bookie checks. The auditor bookie check, checks
ledger metadata to see which bookies should contain entries for each ledger. If
a bookie which should contain entries is unavailable, thea the ledger
containing that entry is marked for recovery. Setting this to 0 disabled the
periodic check. Bookie checks will still run when a bookie fails. The interval
is specified in seconds.
+
+**Default**: 86400
+
+### numAddWorkerThreads
+
+The number of threads that should handle write requests. if zero, the writes
would be handled by netty threads directly.
+
+**Default**: 0
+
+### numReadWorkerThreads
+
+The number of threads that should handle read requests. if zero, the reads
would be handled by netty threads directly.
+
+**Default**: 8
+
+### numHighPriorityWorkerThreads
+
+The umber of threads that should be used for high priority requests (i.e.
recovery reads and adds, and fencing).
+
+**Default**: 8
+
+### maxPendingReadRequestsPerThread
+
+If read workers threads are enabled, limit the number of pending requests, to
avoid the executor queue to grow indefinitely.
+
+**Default**: 2500
+
+### maxPendingAddRequestsPerThread
+
+The limited number of pending requests, which is used to avoid the executor
queue to grow indefinitely when add workers threads are enabled.
+
+**Default**: 10000
+
+### isForceGCAllowWhenNoSpace
+
+Whether force compaction is allowed when the disk is full or almost full.
Forcing GC could get some space back, but could also fill up the disk space
more quickly. This is because new log files are created before GC, while old
garbage log files are deleted after GC.
+
+**Default**: false
+
+### verifyMetadataOnGC
+
+True if the bookie should double check `readMetadata` prior to GC.
+
+**Default**: false
+
+### flushEntrylogBytes
+
+Entry log flush interval in bytes. Flushing in smaller chunks but more
frequently reduces spikes in disk I/O. Flushing too frequently may also affect
performance negatively.
+
+**Default**: 268435456
+
+### readBufferSizeBytes
+
+The number of bytes we should use as capacity for BufferedReadChannel.
+
+**Default**: 4096
+
+### writeBufferSizeBytes
+
+The number of bytes used as capacity for the write buffer
+
+**Default**: 65536
+
+### useHostNameAsBookieID
+
+Whether the bookie should use its hostname to register with the coordination
service (e.g.: zookeeper service). When false, bookie will use its ip address
for the registration.
+
+**Default**: false
+
+### bookieId
+
+If you want to custom a bookie ID or use a dynamic network address for the
bookie, you can set the `bookieId`. <br /><br />Bookie advertises itself using
the `bookieId` rather than the `BookieSocketAddress` (`hostname:port` or
`IP:port`). If you set the `bookieId`, then the `useHostNameAsBookieID` does
not take effect.<br /><br />The `bookieId` is a non-empty string that can
contain ASCII digits and letters ([a-zA-Z9-0]), colons, dashes, and dots. <br
/><br />For more information about ` [...]
+
+**Default**: N/A
+
+### allowEphemeralPorts
+
+Whether the bookie is allowed to use an ephemeral port (port 0) as its server
port. By default, an ephemeral port is not allowed. Using an ephemeral port as
the service port usually indicates a configuration error. However, in unit
tests, using an ephemeral port will address port conflict problems and allow
running tests in parallel.
+
+**Default**: false
+
+### enableLocalTransport
+
+Whether the bookie is allowed to listen for the BookKeeper clients executed on
the local JVM.
+
+**Default**: false
+
+### disableServerSocketBind
+
+Whether the bookie is allowed to disable bind on network interfaces. This
bookie will be available only to BookKeeper clients executed on the local JVM.
+
+**Default**: false
+
+### skipListArenaChunkSize
+
+The number of bytes that we should use as chunk allocation for
`org.apache.bookkeeper.bookie.SkipListArena`.
+
+**Default**: 4194304
+
+### skipListArenaMaxAllocSize
+
+The maximum size that we should allocate from the skiplist arena. Allocations
larger than this should be allocated directly by the VM to avoid fragmentation.
+
+**Default**: 131072
+
+### bookieAuthProviderFactoryClass
+
+The factory class name of the bookie authentication provider. If this is null,
then there is no authentication.
+
+**Default**: null
+
+### statsProviderClass
+
+
[...]
+
+
[...]
+
+### prometheusStatsHttpPort
+
+
[...]
+
+
[...]
+
+### dbStorage_writeCacheMaxSizeMb
+
+Size of Write Cache. Memory is allocated from JVM direct memory. Write cache
is used to buffer entries before flushing into the entry log. For good
performance, it should be big enough to hold a substantial amount of entries in
the flush interval.
+
+**Default**: 25% of direct memory
+
+### dbStorage_readAheadCacheMaxSizeMb
+
+Size of Read cache. Memory is allocated from JVM direct memory. This read
cache is pre-filled doing read-ahead whenever a cache miss happens. By default,
it is allocated to 25% of the available direct memory.
+
+**Default**: N/A
+
+### dbStorage_readAheadCacheBatchSize
+
+How many entries to pre-fill in cache after a read cache miss
+
+**Default**: 1000
+
+### dbStorage_rocksDB_blockCacheSize
+
+Size of RocksDB block-cache. For best performance, this cache should be big
enough to hold a significant portion of the index database which can reach ~2GB
in some cases. By default, it uses 10% of direct memory.
+
+**Default**: N/A
+
+### dbStorage_rocksDB_writeBufferSizeMB
+
+**Default**: 64
+
+### dbStorage_rocksDB_sstSizeInMB
+
+**Default**: 64
+
+### dbStorage_rocksDB_blockSize
+
+**Default**: 65536
+
+### dbStorage_rocksDB_bloomFilterBitsPerKey
+
+**Default**: 10
+
+### dbStorage_rocksDB_numLevels
+
+**Default**: -1
+
+### dbStorage_rocksDB_numFilesInLevel0
+
+**Default**: 4
+
+### dbStorage_rocksDB_maxSizeInLevel1MB
+
+**Default**: 256
diff --git
a/static/reference/3.1.x/config/reference-configuration-log4j-shell.md
b/static/reference/3.1.x/config/reference-configuration-log4j-shell.md
new file mode 100644
index 00000000000..43fca015c0d
--- /dev/null
+++ b/static/reference/3.1.x/config/reference-configuration-log4j-shell.md
@@ -0,0 +1,37 @@
+# Log4j shell
+
+### bookkeeper.root.logger
+
+**Default**: ERROR,CONSOLE
+
+### log4j.rootLogger
+
+**Default**: ${bookkeeper.root.logger}
+
+### log4j.appender.CONSOLE
+
+**Default**: org.apache.log4j.ConsoleAppender
+
+### log4j.appender.CONSOLE.Threshold
+
+**Default**: DEBUG
+
+### log4j.appender.CONSOLE.layout
+
+**Default**: org.apache.log4j.PatternLayout
+
+### log4j.appender.CONSOLE.layout.ConversionPattern
+
+**Default**: %d{ABSOLUTE} %-5p %m%n
+
+### log4j.logger.org.apache.zookeeper
+
+**Default**: ERROR
+
+### log4j.logger.org.apache.bookkeeper
+
+**Default**: ERROR
+
+### log4j.logger.org.apache.bookkeeper.bookie.BookieShell
+
+**Default**: INFO
diff --git a/static/reference/3.1.x/config/reference-configuration-log4j.md
b/static/reference/3.1.x/config/reference-configuration-log4j.md
new file mode 100644
index 00000000000..6aa18e03a23
--- /dev/null
+++ b/static/reference/3.1.x/config/reference-configuration-log4j.md
@@ -0,0 +1,80 @@
+# Log4j
+
+You can set the log level and configuration in the
[log4j2.yaml](https://github.com/apache/pulsar/blob/d557e0aa286866363bc6261dec87790c055db1b0/conf/log4j2.yaml#L155)
file. The following logging configuration parameters are available.
+
+### pulsar.root.logger
+
+**Default**: WARN,CONSOLE
+
+### pulsar.log.dir
+
+**Default**: logs
+
+### pulsar.log.file
+
+**Default**: pulsar.log
+
+### log4j.rootLogger
+
+**Default**: ${pulsar.root.logger}
+
+### log4j.appender.CONSOLE
+
+**Default**: org.apache.log4j.ConsoleAppender
+
+### log4j.appender.CONSOLE.Threshold
+
+**Default**: DEBUG
+
+### log4j.appender.CONSOLE.layout
+
+**Default**: org.apache.log4j.PatternLayout
+
+### log4j.appender.CONSOLE.layout.ConversionPattern
+
+**Default**: %d{ISO8601} - %-5p - [%t:%C{1}@%L] - %m%n
+
+### log4j.appender.ROLLINGFILE
+
+**Default**: org.apache.log4j.DailyRollingFileAppender
+
+### log4j.appender.ROLLINGFILE.Threshold
+
+**Default**: DEBUG
+
+### log4j.appender.ROLLINGFILE.File
+
+**Default**: ${pulsar.log.dir}/${pulsar.log.file}
+
+### log4j.appender.ROLLINGFILE.layout
+
+**Default**: org.apache.log4j.PatternLayout
+
+### log4j.appender.ROLLINGFILE.layout.ConversionPattern
+
+**Default**: %d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
+
+### log4j.appender.TRACEFILE
+
+**Default**: org.apache.log4j.FileAppender
+
+### log4j.appender.TRACEFILE.Threshold
+
+**Default**: TRACE
+
+### log4j.appender.TRACEFILE.File
+
+**Default**: pulsar-trace.log
+
+### log4j.appender.TRACEFILE.layout
+
+**Default**: org.apache.log4j.PatternLayout
+
+### log4j.appender.TRACEFILE.layout.ConversionPattern
+
+**Default**: %d{ISO8601} - %-5p [%t:%C{1}@%L][%x] - %m%n
+
+> Note: 'topic' in log4j2.appender is configurable.
+>
+> - If you want to append all logs to a single topic, set the same topic name.
+> - If you want to append logs to different topics, you can set different
topic names.
diff --git a/static/reference/3.1.x/config/reference-configuration-zookeeper.md
b/static/reference/3.1.x/config/reference-configuration-zookeeper.md
new file mode 100644
index 00000000000..0e4b6b43f54
--- /dev/null
+++ b/static/reference/3.1.x/config/reference-configuration-zookeeper.md
@@ -0,0 +1,83 @@
+# ZooKeeper
+
+ZooKeeper handles a broad range of essential configuration- and
coordination-related tasks for Pulsar. The default configuration file for
ZooKeeper is in the `conf/zookeeper.conf` file in your Pulsar installation. The
following parameters are available:
+
+### tickTime
+
+The tick is the basic unit of time in ZooKeeper, measured in milliseconds and
used to regulate things like heartbeats and timeouts. tickTime is the length of
a single tick.
+
+**Default**: 2000
+
+### initLimit
+
+The maximum time, in ticks, that the leader ZooKeeper server allows follower
ZooKeeper servers to successfully connect and sync. The tick time is set in
milliseconds using the tickTime parameter.
+
+**Default**: 10
+
+### syncLimit
+
+The maximum time, in ticks, that a follower ZooKeeper server is allowed to
sync with other ZooKeeper servers. The tick time is set in milliseconds using
the tickTime parameter.
+
+**Default**: 5
+
+### dataDir
+
+The location where ZooKeeper will store in-memory database snapshots as well
as the transaction log of updates to the database.
+
+**Default**: data/zookeeper
+
+### clientPort
+
+The port on which the ZooKeeper server will listen for connections.
+
+**Default**: 2181
+
+### admin.enableServer
+
+The port at which the admin listens.
+
+**Default**: true
+
+### admin.serverPort
+
+The port at which the admin listens.
+
+**Default**: 9990
+
+### autopurge.snapRetainCount
+
+In ZooKeeper, auto purge determines how many recent snapshots of the database
stored in dataDir to retain within the time interval specified by
autopurge.purgeInterval (while deleting the rest).
+
+**Default**: 3
+
+### autopurge.purgeInterval
+
+The time interval, in hours, by which the ZooKeeper database purge task is
triggered. Setting to a non-zero number will enable auto purge; setting to 0
will disable. Read this guide before enabling auto purge.
+
+**Default**: 1
+
+### forceSync
+
+Requires updates to be synced to media of the transaction log before finishing
processing the update. If this option is set to 'no', ZooKeeper will not
require updates to be synced to the media. WARNING: it's not recommended to run
a production ZK cluster with `forceSync` disabled.
+
+**Default**: yes
+
+### maxClientCnxns
+
+The maximum number of client connections. Increase this if you need to handle
more ZooKeeper clients.
+
+**Default**: 60
+
+---
+
+In addition to the parameters above, configuring ZooKeeper for Pulsar involves
adding a `server.N` line to the `conf/zookeeper.conf` file for each node in the
ZooKeeper cluster, where `N` is the number of the ZooKeeper node. Here's an
example of a three-node ZooKeeper cluster:
+
+```properties
+
+server.1=zk1.us-west.example.com:2888:3888
+server.2=zk2.us-west.example.com:2888:3888
+server.3=zk3.us-west.example.com:2888:3888
+
+```
+
+> We strongly recommend consulting the [ZooKeeper Administrator's
Guide](https://zookeeper.apache.org/doc/current/zookeeperAdmin.html) for a more
thorough and comprehensive introduction to ZooKeeper configuration
diff --git a/static/reference/3.1.x/pulsar-admin/README.md
b/static/reference/3.1.x/pulsar-admin/README.md
new file mode 100644
index 00000000000..63a4c92b5ad
--- /dev/null
+++ b/static/reference/3.1.x/pulsar-admin/README.md
@@ -0,0 +1,16 @@
+`pulsar-admin` is a tool used to administer Pulsar entities.
+
+## Environment variables
+
+You can use the following environment variables to configure `pulsar-admin`.
+
+| Variable | Description | Default
|
+|--------------------------|------------------------------------|------------------|
+| `PULSAR_LOG_CONF` | Log4j configuration file |
conf/log4j2.yaml |
+| `PULSAR_CLIENT_CONF` | Configuration file for the client |
conf/client.conf |
+| `PULSAR_EXTRA_OPTS` | Extra options passed to the JVM | N/A
|
+| `PULSAR_EXTRA_CLASSPATH` | Extra paths for Pulsar's classpath | N/A
|
+
+## Related topics
+
+- [Pulsar admin API](docs/admin-api-overview)
diff --git a/static/reference/3.1.x/pulsar-client/README.md
b/static/reference/3.1.x/pulsar-client/README.md
new file mode 100644
index 00000000000..d25591d2d24
--- /dev/null
+++ b/static/reference/3.1.x/pulsar-client/README.md
@@ -0,0 +1,12 @@
+`pulsar-client` is a tool used to produce and consume messages.
+
+## Environment variables
+
+You can use the following environment variables to configure `pulsar-client`.
+
+| Variable | Description | Default
|
+|--------------------------|------------------------------------|------------------|
+| `PULSAR_LOG_CONF` | Log4j configuration file |
conf/log4j2.yaml |
+| `PULSAR_CLIENT_CONF` | Configuration file for the client |
conf/client.conf |
+| `PULSAR_EXTRA_OPTS` | Extra options passed to the JVM | N/A
|
+| `PULSAR_EXTRA_CLASSPATH` | Extra paths for Pulsar's classpath | N/A
|
diff --git a/static/reference/3.1.x/pulsar-perf/README.md
b/static/reference/3.1.x/pulsar-perf/README.md
new file mode 100644
index 00000000000..3a9ee62e2ef
--- /dev/null
+++ b/static/reference/3.1.x/pulsar-perf/README.md
@@ -0,0 +1,12 @@
+`pulsar-perf` is a tool used to test the performance of Pulsar brokers.
+
+## Environment variables
+
+You can use the following environment variables to configure `pulsar-perf`.
+
+| Variable | Description | Default
|
+|--------------------------|------------------------------------|------------------|
+| `PULSAR_LOG_CONF` | Log4j configuration file |
conf/log4j2.yaml |
+| `PULSAR_CLIENT_CONF` | Configuration file for the client |
conf/client.conf |
+| `PULSAR_EXTRA_OPTS` | Extra options passed to the JVM | N/A
|
+| `PULSAR_EXTRA_CLASSPATH` | Extra paths for Pulsar's classpath | N/A
|
diff --git a/static/reference/3.1.x/pulsar-shell/pulsar-shell.md
b/static/reference/3.1.x/pulsar-shell/pulsar-shell.md
index 1e50f7fe2e6..c7d6e227a09 100644
--- a/static/reference/3.1.x/pulsar-shell/pulsar-shell.md
+++ b/static/reference/3.1.x/pulsar-shell/pulsar-shell.md
@@ -1,6 +1,6 @@
# pulsar-shell
-[Pulsar
shell](https://pulsar.apache.org/docs/next/administration-pulsar-shell) tool.
+[Pulsar shell](pathname:///docs/@pulsar:version@/administration-pulsar-shell)
tool.
### Interactive mode
@@ -10,14 +10,13 @@ Usage
pulsar-shell
```
-| Flag | Description
| Default |
-|--------------------|---------------------------------------------------------------------------|------------------|
-| `-c`, `--config` | Client configuration file. It is used as a `default`
config. | conf/client.conf |
-| `--fail-on-error` | If true, the shell is interrupted when a command throws
an exception. | false |
-| `-h`, `--help` | Show this help.
| |
+| Flag | Description
| Default |
+|-------------------|-----------------------------------------------------------------------|------------------|
+| `-c`, `--config` | Client configuration file. It is used as a `default`
config. | conf/client.conf |
+| `--fail-on-error` | If true, the shell is interrupted when a command throws
an exception. | false |
+| `-h`, `--help` | Show this help.
| |
-
-### Non interactive mode
+### Non-interactive mode
Usage
@@ -27,22 +26,21 @@ pulsar-shell -e [COMMAND]
echo "[COMMAND]" | pulsar-shell -
```
-| Flag | Description
| Default |
-|---------------------------|-----------------------------------------------------------------------------------------------------|-----------------|
-| `-c`, `--config` | Client configuration file. It is used as a
`default` config. | conf/client.conf |
-| `--fail-on-error` | If true, the shell is interrupted when a command
throws an exception. | false |
-| `-np`, `--no-progress` | Display raw output of the commands without the
fancy progress visualization. | false |
-| `-f`, `--filename` | Input filename with a list of commands to be
executed. Each command must be separated by a newline. | |
-| `-e`, `--execute-command` | Execute this command and exit.
| |
-| `-` | Read commands from the standard input.
| |
-| `-h`, `--help` | Show this help.
| |
-
+| Flag | Description
| Default |
+|---------------------------|-----------------------------------------------------------------------------------------------------|------------------|
+| `-c`, `--config` | Client configuration file. It is used as a
`default` config. | conf/client.conf |
+| `--fail-on-error` | If true, the shell is interrupted when a command
throws an exception. | false |
+| `-np`, `--no-progress` | Display raw output of the commands without the
fancy progress visualization. | false |
+| `-f`, `--filename` | Input filename with a list of commands to be
executed. Each command must be separated by a newline. | |
+| `-e`, `--execute-command` | Execute this command and exit.
| |
+| `-` | Read commands from the standard input.
| |
+| `-h`, `--help` | Show this help.
| |
Commands
-* `admin` - See [Admin API](admin-api-overview.md)
-* `client` - See [pulsar-client](#pulsar-client)
-* `config`
+* `admin` - See [Admin
API](pathname:///docs/@pulsar:version@/admin-api-overview)
+* `client` - See [pulsar-client](../pulsar-client/README.md)
+* `config`
### config
@@ -64,11 +62,11 @@ Create a new configuration.
default(localhost)> config create --file ./conf/client.conf mycluster
```
-| Flag | Description | Default |
-|----------|--------------------------|-----------------|
-| `--file` | File path of the config. | |
-| `--url` | URL of the config. | |
-| `--value` | Inline value of the config. Base64-encoded value is supported
with the prefix `base64:`. | |
+| Flag | Description
| Default |
+|-----------|------------------------------------------------------------------------------------------|---------|
+| `--file` | File path of the config.
| |
+| `--url` | URL of the config.
| |
+| `--value` | Inline value of the config. Base64-encoded value is supported
with the prefix `base64:`. | |
#### clone
@@ -78,9 +76,9 @@ Create a new configuration cloning an existing one.
default(localhost)> config clone mycluster --name mycluster2
```
-| Flag | Description | Default |
-|----------|--------------------------|-----------------|
-| `--name` | Name of the new config. | |
+| Flag | Description | Default |
+|----------|-------------------------|---------|
+| `--name` | Name of the new config. | |
#### update
@@ -90,11 +88,11 @@ Update an existing configuration.
default(localhost)> config update --file ./conf/client.conf mycluster
```
-| Flag | Description | Default |
-|----------|--------------------------|-----------------|
-| `--file` | File path of the config. | |
-| `--url` | URL of the config. | |
-| `--value` | Inline value of the config. Base64-encoded value is supported
with the prefix `base64:`. | |
+| Flag | Description
| Default |
+|-----------|------------------------------------------------------------------------------------------|---------|
+| `--file` | File path of the config.
| |
+| `--url` | URL of the config.
| |
+| `--value` | Inline value of the config. Base64-encoded value is supported
with the prefix `base64:`. | |
#### set-property
@@ -104,11 +102,10 @@ Set a value for a specified configuration property.
default(localhost)> config set-property -p webServiceUrl -v
http://<cluster-hostname> mycluster
```
-| Flag | Description | Default |
-|--------------------|-----------------------------|-----------------|
-| `-p`, `--property` | Property name to update. | |
-| `-v`, `--value` | New value for the property. | |
-
+| Flag | Description | Default |
+|--------------------|-----------------------------|---------|
+| `-p`, `--property` | Property name to update. | |
+| `-v`, `--value` | New value for the property. | |
#### get-property
@@ -118,10 +115,9 @@ Get the value for a specified configuration property.
default(localhost)> config get-property -p webServiceUrl mycluster
```
-| Flag | Description | Default |
-|--------------------|-----------------------------|-----------------|
-| `-p`, `--property` | Property name to update. | |
-
+| Flag | Description | Default |
+|--------------------|--------------------------|---------|
+| `-p`, `--property` | Property name to update. | |
#### view
@@ -139,7 +135,6 @@ Delete a config. You can't delete a config if it's
currently used.
default(localhost)> config delete mycluster
```
-
#### list
List all the configuration names.
diff --git a/static/reference/3.1.x/pulsar/README.md
b/static/reference/3.1.x/pulsar/README.md
new file mode 100644
index 00000000000..bf2c5426160
--- /dev/null
+++ b/static/reference/3.1.x/pulsar/README.md
@@ -0,0 +1 @@
+> Docs for `pulsar`.
diff --git a/static/reference/next/cli.md b/static/reference/next/cli.md
index e79763044aa..520d14e5b03 100644
--- a/static/reference/next/cli.md
+++ b/static/reference/next/cli.md
@@ -1,9 +1,3 @@
-Pulsar offers several command-line tools that help you manage Pulsar and
BookKeeper, produce and consume messages, administer Pulsar entities, test
Pulsar, and more.
+# Welcome
-|Goal|Tool
-|---|---
-Manage Pulsar| - `pulsar` <br/><br/> -
[`pulsar-daemon`](/next/pulsar-daemon/pulsar-daemon) <br/><br/> -
[`pulsar-shell`](/next/pulsar-shell/pulsar-shell)
-Produce and consume messages |
[`pulsar-client`](/next/pulsar-client/pulsar-client)
-Administer Pulsar entities | `pulsar-admin`
-Test Pulsar | [`pulsar-perf`](/next/pulsar-perf/pulsar-perf)
-Manage BookKeeper | [`bookkeper`](/next/bookkeeper/bookkeeper)
\ No newline at end of file
+> Welcome to Pulsar CLI Tools Docs.
diff --git a/static/reference/next/client/README.md
b/static/reference/next/client/README.md
index 718f7ad84b7..71df588c6f8 100644
--- a/static/reference/next/client/README.md
+++ b/static/reference/next/client/README.md
@@ -11,12 +11,12 @@ If you create a Java client, producer, consumer, or reader,
you can use the `loa
You can set the client memory allocator configurations through Java
properties.<br />
-| Property | Type | <div style="width: 110pt"> Description </div> | Default |
Available values |
-|---|---|---|---|---|
-|`pulsar.allocator.pooled` | String | If set to `true`, the client uses a
direct memory pool. <br /> If set to `false`, the client uses a heap memory
without pool. | true | <li> true </li> <li> false </li> |
-|`pulsar.allocator.exit_on_oom` | String | Whether to exit the JVM when OOM
happens. | false | <li> true </li> <li> false </li>|
-|`pulsar.allocator.leak_detection` | String | The leak detection policy for
Pulsar bytebuf allocator. <li> **Disabled**: No leak detection and no overhead.
</li> <li> **Simple**: Instruments 1% of the allocated buffer to track for
leaks. </li> <li> **Advanced**: Instruments 1% of the allocated buffer to track
for leaks, reporting stack traces of places where the buffer is used. </li>
<li> **Paranoid**: Instruments 100% of the allocated buffer to track for leaks,
reporting stack traces of [...]
-|`pulsar.allocator.out_of_memory_policy` | String | When an OOM occurs, the
client throws an exception or fallbacks to heap. | FallbackToHeap | <li>
ThrowException </li> <li> FallbackToHeap </li>|
+| Property | Type | <div style="width:
110pt"> Description </div>
[...]
+|-----------------------------------------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
+| `pulsar.allocator.pooled` | String | If set to `true`, the
client uses a direct memory pool. <br /> If set to `false`, the client uses a
heap memory without pool.
[...]
+| `pulsar.allocator.exit_on_oom` | String | Whether to exit the JVM
when OOM happens.
[...]
+| `pulsar.allocator.leak_detection` | String | The leak detection policy
for Pulsar bytebuf allocator. <li> **Disabled**: No leak detection and no
overhead. </li> <li> **Simple**: Instruments 1% of the allocated buffer to
track for leaks. </li> <li> **Advanced**: Instruments 1% of the allocated
buffer to track for leaks, reporting stack traces of places where the buffer is
used. </li> <li> **Paranoid**: Instruments 100% of the allocated buffer to
track for leaks, reporting stack tr [...]
+| `pulsar.allocator.out_of_memory_policy` | String | When an OOM occurs, the
client throws an exception or fallbacks to heap.
[...]
**Example**
@@ -25,4 +25,4 @@ Dpulsar.allocator.pooled=true
Dpulsar.allocator.exit_on_oom=false
Dpulsar.allocator.leak_detection=Disabled
Dpulsar.allocator.out_of_memory_policy=ThrowException
-```
\ No newline at end of file
+```
diff --git a/static/reference/next/config/README.md
b/static/reference/next/config/README.md
index 807c89e69b6..02de3826186 100644
--- a/static/reference/next/config/README.md
+++ b/static/reference/next/config/README.md
@@ -1,6 +1,7 @@
## Pulsar configurations
-You can manage Pulsar configurations through configuration files in the
[`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a
Pulsar installation.
+You can manage Pulsar configurations through configuration files in
+the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a
Pulsar installation.
-
[BookKeeper](/@pulsar:version_reference@/config/reference-configuration-bookkeeper)
- [Broker](/@pulsar:version_reference@/config/reference-configuration-broker)
@@ -14,7 +15,8 @@ You can manage Pulsar configurations through configuration
files in the [`conf`]
### Override client configurations
-If you want to override the configurations of clients internal to brokers,
websockets, and proxies, you can use the following prefix.
+If you want to override the configurations of clients internal to brokers,
websockets, and proxies, you can use the
+following prefix.
| Prefix | Description
|
|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
@@ -23,21 +25,23 @@ If you want to override the configurations of clients
internal to brokers, webso
> Notes:
> * This override feature only applies to Pulsar 2.10.1 and later versions.
-> * When running the function worker within the broker, you have to configure
those clients by using the `functions_worker.yml` file. These prefixed
configurations do not apply to any of those clients.
+> * When running the function worker within the broker, you have to configure
those clients by using
+ the `functions_worker.yml` file. These prefixed configurations do not
apply to any of those clients.
### Set specific configurations using Java system property
You can set specific configurations through Java properties.
-| **Property** | **Description**
|
-| ------------------ |
------------------------------------------------------------ |
+| **Property** | **Description**
|
+|--------------------|---------------------------------------------------------------------------------------------------------------|
| pulsar.enableUring | Use `io_uring` instead of `epoll` as network IO mode.
`-Dpulsar.enableUring=1` means this feature is enabled. |
## Broker load balancing configurations
-Below is a brief summary of configurations for broker load balancing.
+Below is a brief summary of configurations for broker load balancing.
-For detailed descriptions of each configuration, see [Broker load balancing |
Configurations](pathname:///reference/#/@pulsar:version_reference@/config/reference-configuration-broker).
+For detailed descriptions of each configuration,
+see [Broker load balancing |
Configurations](pathname:///reference/#/@pulsar:version_reference@/config/reference-configuration-broker).
> Note
> Configurations with an asterisk (*) are only available in the extensible
> load balancer.
@@ -53,14 +57,13 @@ For detailed descriptions of each configuration, see
[Broker load balancing | Co
- loadBalancerMemoryResourceWeight
- loadBalancerDirectMemoryResourceWeight
- loadBalancerHistoryResourcePercentage
-
+
### TopK bundle load data
- loadBalancerReportUpdateMinIntervalMillis
- statsUpdateFrequencyInSecs
- loadBalancerMaxNumberOfBundlesInBundleLoadReport*
-
### Bundle-broker assignment
- loadBalancerAverageResourceUsageDifferenceThresholdPercentage
@@ -76,11 +79,13 @@ For detailed descriptions of each configuration, see
[Broker load balancing | Co
- loadBalancerSheddingIntervalMinutes
- loadBalancerSheddingGracePeriodMinutes
- loadBalancerBrokerOverloadedThresholdPercentage
-- loadBalancerLoadSheddingStrategy. The default value is
[org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder](https://github.com/apache/pulsar/blob/782e91fe327efe2c9c9107d6c679c2837d43935b/conf/broker.conf#L1324).
Available values are:
- - `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
- - `org.apache.pulsar.broker.loadbalance.impl.UniformLoadShedder`
- - `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder`
- -
`org.apache.pulsar.broker.loadbalance.extensions.scheduler.TransferShedder`
+- loadBalancerLoadSheddingStrategy. The default value
+ is
[org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder](https://github.com/apache/pulsar/blob/782e91fe327efe2c9c9107d6c679c2837d43935b/conf/broker.conf#L1324).
+ Available values are:
+ - `org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder`
+ - `org.apache.pulsar.broker.loadbalance.impl.UniformLoadShedder`
+ - `org.apache.pulsar.broker.loadbalance.impl.OverloadShedder`
+ - `org.apache.pulsar.broker.loadbalance.extensions.scheduler.TransferShedder`
- loadBalancerTransferEnabled*
- loadBalancerBrokerLoadTargetStd*
- loadBalancerSheddingConditionHitCountThreshold*
@@ -100,4 +105,4 @@ For detailed descriptions of each configuration, see
[Broker load balancing | Co
- loadBalancerNamespaceMaximumBundles
- loadBalancerSplitIntervalMinutes*
- loadBalancerNamespaceBundleSplitConditionHitCountThreshold*
-- loadBalancerMaxNumberOfBundlesToSplitPerCycle*
\ No newline at end of file
+- loadBalancerMaxNumberOfBundlesToSplitPerCycle*
diff --git a/static/reference/next/pulsar-admin/README.md
b/static/reference/next/pulsar-admin/README.md
index 3180c0ef14d..63a4c92b5ad 100644
--- a/static/reference/next/pulsar-admin/README.md
+++ b/static/reference/next/pulsar-admin/README.md
@@ -4,13 +4,13 @@
You can use the following environment variables to configure `pulsar-admin`.
-|Variable|Description|Default|
-|---|---|---|
-|`PULSAR_LOG_CONF`|Log4j configuration file|conf/log4j2.yaml|
-|`PULSAR_CLIENT_CONF`|Configuration file for the client|conf/client.conf|
-|`PULSAR_EXTRA_OPTS`|Extra options passed to the JVM|N/A|
-|`PULSAR_EXTRA_CLASSPATH`|Extra paths for Pulsar's classpath|N/A|
+| Variable | Description | Default
|
+|--------------------------|------------------------------------|------------------|
+| `PULSAR_LOG_CONF` | Log4j configuration file |
conf/log4j2.yaml |
+| `PULSAR_CLIENT_CONF` | Configuration file for the client |
conf/client.conf |
+| `PULSAR_EXTRA_OPTS` | Extra options passed to the JVM | N/A
|
+| `PULSAR_EXTRA_CLASSPATH` | Extra paths for Pulsar's classpath | N/A
|
## Related topics
-- [Pulsar admin API](docs/admin-api-overview)
\ No newline at end of file
+- [Pulsar admin API](docs/admin-api-overview)
diff --git a/static/reference/next/pulsar-client/README.md
b/static/reference/next/pulsar-client/README.md
index f875e02f284..d25591d2d24 100644
--- a/static/reference/next/pulsar-client/README.md
+++ b/static/reference/next/pulsar-client/README.md
@@ -4,9 +4,9 @@
You can use the following environment variables to configure `pulsar-client`.
-|Variable|Description|Default|
-|---|---|---|
-|`PULSAR_LOG_CONF`|Log4j configuration file|conf/log4j2.yaml|
-|`PULSAR_CLIENT_CONF`|Configuration file for the client|conf/client.conf|
-|`PULSAR_EXTRA_OPTS`|Extra options passed to the JVM|N/A|
-|`PULSAR_EXTRA_CLASSPATH`|Extra paths for Pulsar's classpath|N/A|
\ No newline at end of file
+| Variable | Description | Default
|
+|--------------------------|------------------------------------|------------------|
+| `PULSAR_LOG_CONF` | Log4j configuration file |
conf/log4j2.yaml |
+| `PULSAR_CLIENT_CONF` | Configuration file for the client |
conf/client.conf |
+| `PULSAR_EXTRA_OPTS` | Extra options passed to the JVM | N/A
|
+| `PULSAR_EXTRA_CLASSPATH` | Extra paths for Pulsar's classpath | N/A
|
diff --git a/static/reference/next/pulsar-perf/README.md
b/static/reference/next/pulsar-perf/README.md
index 6359464dd35..3a9ee62e2ef 100644
--- a/static/reference/next/pulsar-perf/README.md
+++ b/static/reference/next/pulsar-perf/README.md
@@ -4,9 +4,9 @@
You can use the following environment variables to configure `pulsar-perf`.
-|Variable|Description|Default|
-|---|---|---|
-|`PULSAR_LOG_CONF`|Log4j configuration file|conf/log4j2.yaml|
-|`PULSAR_CLIENT_CONF`|Configuration file for the client|conf/client.conf|
-|`PULSAR_EXTRA_OPTS`|Extra options passed to the JVM|N/A|
-|`PULSAR_EXTRA_CLASSPATH`|Extra paths for Pulsar's classpath|N/A|
\ No newline at end of file
+| Variable | Description | Default
|
+|--------------------------|------------------------------------|------------------|
+| `PULSAR_LOG_CONF` | Log4j configuration file |
conf/log4j2.yaml |
+| `PULSAR_CLIENT_CONF` | Configuration file for the client |
conf/client.conf |
+| `PULSAR_EXTRA_OPTS` | Extra options passed to the JVM | N/A
|
+| `PULSAR_EXTRA_CLASSPATH` | Extra paths for Pulsar's classpath | N/A
|
diff --git a/static/reference/next/pulsar-shell/pulsar-shell.md
b/static/reference/next/pulsar-shell/pulsar-shell.md
index 1e50f7fe2e6..c7d6e227a09 100644
--- a/static/reference/next/pulsar-shell/pulsar-shell.md
+++ b/static/reference/next/pulsar-shell/pulsar-shell.md
@@ -1,6 +1,6 @@
# pulsar-shell
-[Pulsar
shell](https://pulsar.apache.org/docs/next/administration-pulsar-shell) tool.
+[Pulsar shell](pathname:///docs/@pulsar:version@/administration-pulsar-shell)
tool.
### Interactive mode
@@ -10,14 +10,13 @@ Usage
pulsar-shell
```
-| Flag | Description
| Default |
-|--------------------|---------------------------------------------------------------------------|------------------|
-| `-c`, `--config` | Client configuration file. It is used as a `default`
config. | conf/client.conf |
-| `--fail-on-error` | If true, the shell is interrupted when a command throws
an exception. | false |
-| `-h`, `--help` | Show this help.
| |
+| Flag | Description
| Default |
+|-------------------|-----------------------------------------------------------------------|------------------|
+| `-c`, `--config` | Client configuration file. It is used as a `default`
config. | conf/client.conf |
+| `--fail-on-error` | If true, the shell is interrupted when a command throws
an exception. | false |
+| `-h`, `--help` | Show this help.
| |
-
-### Non interactive mode
+### Non-interactive mode
Usage
@@ -27,22 +26,21 @@ pulsar-shell -e [COMMAND]
echo "[COMMAND]" | pulsar-shell -
```
-| Flag | Description
| Default |
-|---------------------------|-----------------------------------------------------------------------------------------------------|-----------------|
-| `-c`, `--config` | Client configuration file. It is used as a
`default` config. | conf/client.conf |
-| `--fail-on-error` | If true, the shell is interrupted when a command
throws an exception. | false |
-| `-np`, `--no-progress` | Display raw output of the commands without the
fancy progress visualization. | false |
-| `-f`, `--filename` | Input filename with a list of commands to be
executed. Each command must be separated by a newline. | |
-| `-e`, `--execute-command` | Execute this command and exit.
| |
-| `-` | Read commands from the standard input.
| |
-| `-h`, `--help` | Show this help.
| |
-
+| Flag | Description
| Default |
+|---------------------------|-----------------------------------------------------------------------------------------------------|------------------|
+| `-c`, `--config` | Client configuration file. It is used as a
`default` config. | conf/client.conf |
+| `--fail-on-error` | If true, the shell is interrupted when a command
throws an exception. | false |
+| `-np`, `--no-progress` | Display raw output of the commands without the
fancy progress visualization. | false |
+| `-f`, `--filename` | Input filename with a list of commands to be
executed. Each command must be separated by a newline. | |
+| `-e`, `--execute-command` | Execute this command and exit.
| |
+| `-` | Read commands from the standard input.
| |
+| `-h`, `--help` | Show this help.
| |
Commands
-* `admin` - See [Admin API](admin-api-overview.md)
-* `client` - See [pulsar-client](#pulsar-client)
-* `config`
+* `admin` - See [Admin
API](pathname:///docs/@pulsar:version@/admin-api-overview)
+* `client` - See [pulsar-client](../pulsar-client/README.md)
+* `config`
### config
@@ -64,11 +62,11 @@ Create a new configuration.
default(localhost)> config create --file ./conf/client.conf mycluster
```
-| Flag | Description | Default |
-|----------|--------------------------|-----------------|
-| `--file` | File path of the config. | |
-| `--url` | URL of the config. | |
-| `--value` | Inline value of the config. Base64-encoded value is supported
with the prefix `base64:`. | |
+| Flag | Description
| Default |
+|-----------|------------------------------------------------------------------------------------------|---------|
+| `--file` | File path of the config.
| |
+| `--url` | URL of the config.
| |
+| `--value` | Inline value of the config. Base64-encoded value is supported
with the prefix `base64:`. | |
#### clone
@@ -78,9 +76,9 @@ Create a new configuration cloning an existing one.
default(localhost)> config clone mycluster --name mycluster2
```
-| Flag | Description | Default |
-|----------|--------------------------|-----------------|
-| `--name` | Name of the new config. | |
+| Flag | Description | Default |
+|----------|-------------------------|---------|
+| `--name` | Name of the new config. | |
#### update
@@ -90,11 +88,11 @@ Update an existing configuration.
default(localhost)> config update --file ./conf/client.conf mycluster
```
-| Flag | Description | Default |
-|----------|--------------------------|-----------------|
-| `--file` | File path of the config. | |
-| `--url` | URL of the config. | |
-| `--value` | Inline value of the config. Base64-encoded value is supported
with the prefix `base64:`. | |
+| Flag | Description
| Default |
+|-----------|------------------------------------------------------------------------------------------|---------|
+| `--file` | File path of the config.
| |
+| `--url` | URL of the config.
| |
+| `--value` | Inline value of the config. Base64-encoded value is supported
with the prefix `base64:`. | |
#### set-property
@@ -104,11 +102,10 @@ Set a value for a specified configuration property.
default(localhost)> config set-property -p webServiceUrl -v
http://<cluster-hostname> mycluster
```
-| Flag | Description | Default |
-|--------------------|-----------------------------|-----------------|
-| `-p`, `--property` | Property name to update. | |
-| `-v`, `--value` | New value for the property. | |
-
+| Flag | Description | Default |
+|--------------------|-----------------------------|---------|
+| `-p`, `--property` | Property name to update. | |
+| `-v`, `--value` | New value for the property. | |
#### get-property
@@ -118,10 +115,9 @@ Get the value for a specified configuration property.
default(localhost)> config get-property -p webServiceUrl mycluster
```
-| Flag | Description | Default |
-|--------------------|-----------------------------|-----------------|
-| `-p`, `--property` | Property name to update. | |
-
+| Flag | Description | Default |
+|--------------------|--------------------------|---------|
+| `-p`, `--property` | Property name to update. | |
#### view
@@ -139,7 +135,6 @@ Delete a config. You can't delete a config if it's
currently used.
default(localhost)> config delete mycluster
```
-
#### list
List all the configuration names.
diff --git a/tools/pytools/bin/reference-doc-generator.py
b/tools/pytools/bin/reference-doc-generator.py
index 59a0e091c41..e8293bfec14 100755
--- a/tools/pytools/bin/reference-doc-generator.py
+++ b/tools/pytools/bin/reference-doc-generator.py
@@ -18,11 +18,13 @@
# under the License.
import enum
+import shutil
from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
from pathlib import Path
import semver
+from constant import site_path
from execute import config_doc_generator, pulsar_admin_clidoc_generator,
pulsar_clidoc_generator
from execute import pulsar_client_clidoc_generator,
pulsar_perf_clidoc_generator
@@ -45,14 +47,19 @@ if __name__ == '__main__':
args = parser.parse_args()
master_path = Path(args.master_path)
kinds = set(args.kind)
- if Kind.all in kinds:
- kinds = {Kind.config, Kind.admin, Kind.pulsar, Kind.client, Kind.perf}
+ version = args.version
- if args.version != 'next':
- ver = semver.VersionInfo.parse(args.version)
+ if version != 'next':
+ ver = semver.VersionInfo.parse(version)
version = f"{ver.major}.{ver.minor}.x"
- else:
- version = args.version
+
+ if version != 'next' and Kind.all in kinds:
+ src = site_path() / 'static' / 'reference' / 'next'
+ dst = site_path() / 'static' / 'reference' / version
+ shutil.copytree(src, dst)
+
+ if Kind.all in kinds:
+ kinds = {Kind.config, Kind.admin, Kind.pulsar, Kind.client, Kind.perf}
for kind in kinds:
if kind == Kind.config:
