sijie closed pull request #1948: Documentation for threshold based compaction
URL: https://github.com/apache/incubator-pulsar/pull/1948
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/site/_data/cli/pulsar-admin.yaml b/site/_data/cli/pulsar-admin.yaml
index 9470d7c07c..bfae4eadbe 100644
--- a/site/_data/cli/pulsar-admin.yaml
+++ b/site/_data/cli/pulsar-admin.yaml
@@ -459,6 +459,16 @@ commands:
description: # `{start_boundary}_{end_boundary}`
- flags: -s, --sub
description: The subscription name
+ - name: get-compaction-threshold
+ description: Get compactionThreshold for a namespace
+ argument: tenant/namespace
+ - name: set-compaction-threshold
+ description: Set compactionThreshold for a namespace
+ argument: tenant/namespace
+ options:
+ - flags: -t, --threshold
+ description: "Maximum number of bytes in a topic backlog before
compaction is triggered (eg: 10M, 16G, 3T). 0 disables automatic compaction"
+ default: '0'
- name: ns-isolation-policy
description: Operations for managing namespace isolation policies.
subcommands:
diff --git a/site/docs/latest/cookbooks/compaction.md
b/site/docs/latest/cookbooks/compaction.md
index 261327e260..3b11400ef8 100644
--- a/site/docs/latest/cookbooks/compaction.md
+++ b/site/docs/latest/cookbooks/compaction.md
@@ -8,7 +8,7 @@ Pulsar's [topic
compaction](../../getting-started/ConceptsAndArchitecture#compac
To use compaction:
* You need to give messages keys, as topic compaction in Pulsar takes place on
a *per-key basis* (i.e. messages are compacted based on their key). For a stock
ticker use case, the stock symbol---e.g. `AAPL` or `GOOG`---could serve as the
key (more on this [below](#when)). Messages without keys will be left alone by
the compaction process.
-* You must manually [trigger](#trigger) compaction using the Pulsar
administrative API. This will both run a compaction operation *and* mark the
topic as a compacted topic.
+* Compaction can be configured to run [automatically](#automatic), or you can
manually [trigger](#trigger) compaction using the Pulsar administrative API.
* Your {% popover consumers %} must be [configured](#config) to read from
compacted topics ([Java consumers](#java), for example, have a `readCompacted`
setting that must be set to `true`). If this configuration is not set,
consumers will still be able to read from the non-compacted topic.
## When should I use compacted topics? {#when}
@@ -19,7 +19,20 @@ The classic example of a topic that could benefit from
compaction would be a sto
{% include admonition.html type="warning" content="Compaction only works on
topics where each message has a key (as in the stock ticker example, where the
stock symbol serves as the key). Keys can be thought of as the axis along which
compaction is applied." %}
-## Triggering compaction {#trigger}
+## Configuring compaction to run automatically {#automatic}
+
+Tenant administrators can configure a policy for compaction at the namespace
level. The policy specifies how large the topic backlog can grow before
compaction is triggered.
+
+For example, to trigger compaction when the backlog reaches 100MB:
+
+```bash
+$ bin/pulsar-admin namespaces set-compaction-threshold \
+ --threshold 100M my-tenant/my-namespace
+```
+
+Configuring the compaction threshold on a namespace will apply to all topics
within that namespace.
+
+## Triggering compaction manually {#trigger}
In order to run compaction on a topic, you need to use the [`topics
compact`](../../CliTools#pulsar-admin-topics-compact) command for the
[`pulsar-admin`](../../CliTools#pulsar-admin) CLI tool. Here's an example:
@@ -51,8 +64,6 @@ $ bin/pulsar compact-topic \
How often you [trigger compaction](#trigger) will vary widely based on the use
case. If you want a compacted topic to be extremely speedy on read, then you
should run compaction fairly frequently.
-{% include admonition.html type="warning" title="No automatic compaction"
content="Currently, all topic compaction in Pulsar must be initiated manually
via the [CLI](#trigger) or [REST API](../../reference/RestApi)." %}
-
## Consumer configuration {#config}
Pulsar consumers and readers need to be configured to read from compacted
topics. The sections below show you how to enable compacted topic reads for
Pulsar's language clients. If the
diff --git a/site/docs/latest/getting-started/ConceptsAndArchitecture.md
b/site/docs/latest/getting-started/ConceptsAndArchitecture.md
index 6fcad5912a..d6efb4cd5f 100644
--- a/site/docs/latest/getting-started/ConceptsAndArchitecture.md
+++ b/site/docs/latest/getting-started/ConceptsAndArchitecture.md
@@ -553,7 +553,7 @@ Pulsar's topic compaction feature:
* Allos for much more efficient "rewind" through topic logs
* Applies only to [persistent topics](#persistent-storage)
-* Is triggered manually via the command line. See the [Topic compaction
cookbook](../../cookbooks/compaction)
+* Triggered automatically when the backlog reaches a certain size or can be
triggered manually via the command line. See the [Topic compaction
cookbook](../../cookbooks/compaction)
* Is conceptually and operationally distinct from [retention and
expiry](#message-retention-and-expiry). Topic compaction *does*, however,
respect retention. If retention has removed a message from the message backlog
of a topic, the message will also not be readable from the compacted topic
ledger.
{% include admonition.html type="info" title="Topic compaction example: the
stock ticker"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
