This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 673db8b Documentation for threshold based compaction (#1948)
673db8b is described below
commit 673db8bf4d74ee11bf6797edcc4f7227f54a0cb5
Author: Ivan Kelly <[email protected]>
AuthorDate: Mon Jun 11 19:46:27 2018 +0200
Documentation for threshold based compaction (#1948)
Updates for reference and cookbook documentation for compaction to
show users how to set up automatic compaction based on a size threshold.
---
site/_data/cli/pulsar-admin.yaml | 10 ++++++++++
site/docs/latest/cookbooks/compaction.md | 19 +++++++++++++++----
.../latest/getting-started/ConceptsAndArchitecture.md | 2 +-
3 files changed, 26 insertions(+), 5 deletions(-)
diff --git a/site/_data/cli/pulsar-admin.yaml b/site/_data/cli/pulsar-admin.yaml
index 9470d7c..bfae4ea 100644
--- a/site/_data/cli/pulsar-admin.yaml
+++ b/site/_data/cli/pulsar-admin.yaml
@@ -459,6 +459,16 @@ commands:
description: # `{start_boundary}_{end_boundary}`
- flags: -s, --sub
description: The subscription name
+ - name: get-compaction-threshold
+ description: Get compactionThreshold for a namespace
+ argument: tenant/namespace
+ - name: set-compaction-threshold
+ description: Set compactionThreshold for a namespace
+ argument: tenant/namespace
+ options:
+ - flags: -t, --threshold
+ description: "Maximum number of bytes in a topic backlog before
compaction is triggered (eg: 10M, 16G, 3T). 0 disables automatic compaction"
+ default: '0'
- name: ns-isolation-policy
description: Operations for managing namespace isolation policies.
subcommands:
diff --git a/site/docs/latest/cookbooks/compaction.md
b/site/docs/latest/cookbooks/compaction.md
index 261327e..3b11400 100644
--- a/site/docs/latest/cookbooks/compaction.md
+++ b/site/docs/latest/cookbooks/compaction.md
@@ -8,7 +8,7 @@ Pulsar's [topic
compaction](../../getting-started/ConceptsAndArchitecture#compac
To use compaction:
* You need to give messages keys, as topic compaction in Pulsar takes place on
a *per-key basis* (i.e. messages are compacted based on their key). For a stock
ticker use case, the stock symbol---e.g. `AAPL` or `GOOG`---could serve as the
key (more on this [below](#when)). Messages without keys will be left alone by
the compaction process.
-* You must manually [trigger](#trigger) compaction using the Pulsar
administrative API. This will both run a compaction operation *and* mark the
topic as a compacted topic.
+* Compaction can be configured to run [automatically](#automatic), or you can
manually [trigger](#trigger) compaction using the Pulsar administrative API.
* Your {% popover consumers %} must be [configured](#config) to read from
compacted topics ([Java consumers](#java), for example, have a `readCompacted`
setting that must be set to `true`). If this configuration is not set,
consumers will still be able to read from the non-compacted topic.
## When should I use compacted topics? {#when}
@@ -19,7 +19,20 @@ The classic example of a topic that could benefit from
compaction would be a sto
{% include admonition.html type="warning" content="Compaction only works on
topics where each message has a key (as in the stock ticker example, where the
stock symbol serves as the key). Keys can be thought of as the axis along which
compaction is applied." %}
-## Triggering compaction {#trigger}
+## Configuring compaction to run automatically {#automatic}
+
+Tenant administrators can configure a policy for compaction at the namespace
level. The policy specifies how large the topic backlog can grow before
compaction is triggered.
+
+For example, to trigger compaction when the backlog reaches 100MB:
+
+```bash
+$ bin/pulsar-admin namespaces set-compaction-threshold \
+ --threshold 100M my-tenant/my-namespace
+```
+
+Configuring the compaction threshold on a namespace will apply to all topics
within that namespace.
+
+## Triggering compaction manually {#trigger}
In order to run compaction on a topic, you need to use the [`topics
compact`](../../CliTools#pulsar-admin-topics-compact) command for the
[`pulsar-admin`](../../CliTools#pulsar-admin) CLI tool. Here's an example:
@@ -51,8 +64,6 @@ $ bin/pulsar compact-topic \
How often you [trigger compaction](#trigger) will vary widely based on the use
case. If you want a compacted topic to be extremely speedy on read, then you
should run compaction fairly frequently.
-{% include admonition.html type="warning" title="No automatic compaction"
content="Currently, all topic compaction in Pulsar must be initiated manually
via the [CLI](#trigger) or [REST API](../../reference/RestApi)." %}
-
## Consumer configuration {#config}
Pulsar consumers and readers need to be configured to read from compacted
topics. The sections below show you how to enable compacted topic reads for
Pulsar's language clients. If the
diff --git a/site/docs/latest/getting-started/ConceptsAndArchitecture.md
b/site/docs/latest/getting-started/ConceptsAndArchitecture.md
index 6fcad59..d6efb4c 100644
--- a/site/docs/latest/getting-started/ConceptsAndArchitecture.md
+++ b/site/docs/latest/getting-started/ConceptsAndArchitecture.md
@@ -553,7 +553,7 @@ Pulsar's topic compaction feature:
* Allos for much more efficient "rewind" through topic logs
* Applies only to [persistent topics](#persistent-storage)
-* Is triggered manually via the command line. See the [Topic compaction
cookbook](../../cookbooks/compaction)
+* Triggered automatically when the backlog reaches a certain size or can be
triggered manually via the command line. See the [Topic compaction
cookbook](../../cookbooks/compaction)
* Is conceptually and operationally distinct from [retention and
expiry](#message-retention-and-expiry). Topic compaction *does*, however,
respect retention. If retention has removed a message from the message backlog
of a topic, the message will also not be readable from the compacted topic
ledger.
{% include admonition.html type="info" title="Topic compaction example: the
stock ticker"
--
To stop receiving notification emails like this one, please contact
[email protected].
