This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 9ae78a4 Update doc for support set/get/remove message deduplication
policy at topic level (#7918)
9ae78a4 is described below
commit 9ae78a403bd97c24dc467e19ebb30e47d3e9fa6c
Author: HuanliMeng <[email protected]>
AuthorDate: Fri Aug 28 21:43:47 2020 +0800
Update doc for support set/get/remove message deduplication policy at topic
level (#7918)
### Motivation
PR (https://github.com/apache/pulsar/pull/7821) supports set/get/remove
the deduplication policy at the topic level.
### Modifications
Update the doc based on code updates.
The following docs are updated:
- Messaging: message deduplication
- Manage message deduplication
- pulsar admin cli reference
---
site2/docs/concepts-messaging.md | 4 ++--
site2/docs/cookbooks-deduplication.md | 19 ++++++++++++-------
site2/docs/reference-pulsar-admin.md | 32 ++++++++++++++++++++++++++++++++
3 files changed, 46 insertions(+), 9 deletions(-)
diff --git a/site2/docs/concepts-messaging.md b/site2/docs/concepts-messaging.md
index 8a761e4..2a1e782 100644
--- a/site2/docs/concepts-messaging.md
+++ b/site2/docs/concepts-messaging.md
@@ -458,7 +458,7 @@ With message expiry, shown at the bottom, some messages are
<span style="color:
## Message deduplication
-Message duplication occurs when a message is
persisted](concepts-architecture-overview.md#persistent-storage) by Pulsar more
than once. Message deduplication is an optional Pulsar feature that prevents
unnecessary message duplication by processing each message only once, even if
the message is received more than once.
+Message duplication occurs when a message is
[persisted](concepts-architecture-overview.md#persistent-storage) by Pulsar
more than once. Message deduplication is an optional Pulsar feature that
prevents unnecessary message duplication by processing each message only once,
even if the message is received more than once.
The following diagram illustrates what happens when message deduplication is
disabled vs. enabled:
@@ -469,7 +469,7 @@ Message deduplication is disabled in the scenario shown at
the top. Here, a prod
In the second scenario at the bottom, the producer publishes message 1, which
is received by the broker and persisted, as in the first scenario. When the
producer attempts to publish the message again, however, the broker knows that
it has already seen message 1 and thus does not persist the message.
-> Message deduplication is handled at the namespace level. For more
instructions, see the [message deduplication
cookbook](cookbooks-deduplication.md).
+> Message deduplication is handled at the namespace level or the topic level.
For more instructions, see the [message deduplication
cookbook](cookbooks-deduplication.md).
### Producer idempotency
diff --git a/site2/docs/cookbooks-deduplication.md
b/site2/docs/cookbooks-deduplication.md
index 58b3a3f..6140bde 100644
--- a/site2/docs/cookbooks-deduplication.md
+++ b/site2/docs/cookbooks-deduplication.md
@@ -10,10 +10,11 @@ To use message deduplication in Pulsar, you need to
configure your Pulsar broker
## How it works
-You can enable or disable message deduplication on a per-namespace basis. By
default, it is disabled on all namespaces. You can enable it in the following
ways:
+You can enable or disable message deduplication at the namespace level or the
topic level. By default, it is disabled on all namespaces or topics. You can
enable it in the following ways:
-* Enable for all namespaces at the broker-level
-* Enable for specific namespaces with the `pulsar-admin namespaces` interface
+* Enable deduplication for all namespaces/topics at the broker-level.
+* Enable deduplication for a specific namespace with the `pulsar-admin
namespaces` interface.
+* Enable deduplication for a specific topic with the `pulsar-admin topics`
interface.
## Configure message deduplication
@@ -21,20 +22,22 @@ You can configure message deduplication in Pulsar using the
[`broker.conf`](refe
Parameter | Description | Default
:---------|:------------|:-------
-`brokerDeduplicationEnabled` | Sets the default behavior for message
deduplication in the Pulsar broker. If it is set to `true`, message
deduplication is enabled by default on all namespaces; if it is set to `false`,
you have to enable or disable deduplication on a per-namespace basis. | `false`
+`brokerDeduplicationEnabled` | Sets the default behavior for message
deduplication in the Pulsar broker. If it is set to `true`, message
deduplication is enabled on all namespaces/topics. If it is set to `false`, you
have to enable or disable deduplication at the namespace level or the topic
level. | `false`
`brokerDeduplicationMaxNumberOfProducers` | The maximum number of producers
for which information is stored for deduplication purposes. | `10000`
`brokerDeduplicationEntriesInterval` | The number of entries after which a
deduplication informational snapshot is taken. A larger interval leads to fewer
snapshots being taken, though this lengthens the topic recovery time (the time
required for entries published after the snapshot to be replayed). | `1000`
`brokerDeduplicationProducerInactivityTimeoutMinutes` | The time of inactivity
(in minutes) after which the broker discards deduplication information related
to a disconnected producer. | `360` (6 hours)
### Set default value at the broker-level
-By default, message deduplication is *disabled* on all Pulsar namespaces. To
enable it by default on all namespaces, set the `brokerDeduplicationEnabled`
parameter to `true` and re-start the broker.
+By default, message deduplication is *disabled* on all Pulsar
namespaces/topics. To enable it on all namespaces/topics, set the
`brokerDeduplicationEnabled` parameter to `true` and re-start the broker.
Even if you set the value for `brokerDeduplicationEnabled`, enabling or
disabling via Pulsar admin CLI overrides the default settings at the
broker-level.
### Enable message deduplication
-Though message deduplication is disabled by default at broker-level, you can
enable message deduplication for specific namespaces using the [`pulsar-admin
namespace
set-deduplication`](reference-pulsar-admin.md#namespace-set-deduplication)
command. You can use the `--enable`/`-e` flag and specify the namespace. The
following is an example with `<tenant>/<namespace>`:
+Though message deduplication is disabled by default at the broker level, you
can enable message deduplication for a specific namespace or topic using the
[`pulsar-admin namespaces
set-deduplication`](reference-pulsar-admin.md#namespace-set-deduplication) or
the [`pulsar-admin topics
set-deduplication`](reference-pulsar-admin.md#topic-set-deduplication) command.
You can use the `--enable`/`-e` flag and specify the namespace/topic.
+
+The following example shows how to enable message deduplication at the
namespace level.
```bash
$ bin/pulsar-admin namespaces set-deduplication \
@@ -44,7 +47,9 @@ $ bin/pulsar-admin namespaces set-deduplication \
### Disable message deduplication
-Even if you enable message deduplication at broker-level, you can disable
message deduplication for a specific namespace using the [`pulsar-admin
namespace
set-deduplication`](reference-pulsar-admin.md#namespace-set-deduplication)
command. Use the `--disable`/`-d` flag and specify the namespace. The following
is an example with `<tenant>/<namespace>`:
+Even if you enable message deduplication at the broker level, you can disable
message deduplication for a specific namespace or topic using the
[`pulsar-admin namespace
set-deduplication`](reference-pulsar-admin.md#namespace-set-deduplication) or
the [`pulsar-admin topics
set-deduplication`](reference-pulsar-admin.md#topic-set-deduplication) command.
Use the `--disable`/`-d` flag and specify the namespace/topic.
+
+The following example shows how to disable message deduplication at the
namespace level.
```bash
$ bin/pulsar-admin namespaces set-deduplication \
diff --git a/site2/docs/reference-pulsar-admin.md
b/site2/docs/reference-pulsar-admin.md
index b37d84a..9c583d5 100644
--- a/site2/docs/reference-pulsar-admin.md
+++ b/site2/docs/reference-pulsar-admin.md
@@ -1787,6 +1787,9 @@ Subcommands
* `reset-cursor`
* `get-message-by-id`
* `last-message-id`
+* `get-deduplication`
+* `set-deduplication`
+* `remove-deduplication`
### `compact`
Run compaction on the specified topic (persistent topics only)
@@ -2202,6 +2205,35 @@ Options
|`-l`, `--ledgerId`|The ledger id |0|
|`-e`, `--entryId`|The entry id |0|
+### `get-deduplication`
+Get a deduplication policy for a topic.
+
+Usage
+```bash
+$ pulsar-admin topics get-deduplication tenant/namespace/topic
+```
+
+### `set-deduplication`
+Enable or disable message deduplication on a topic.
+
+Usage
+```bash
+$ pulsar-admin topics set-deduplication tenant/namespace/topic
+```
+
+Options
+|Flag|Description|Default|
+|---|---|---|
+|`--enable`, `-e`|Enable message deduplication on the specified topic.|false|
+|`--disable`, `-d`|Disable message deduplication on the specified topic.|false|
+
+### `remove-deduplication`
+Remove a deduplication policy from a topic.
+
+Usage
+```bash
+$ pulsar-admin topics remove-deduplication tenant/namespace/topic
+```
## `tenants`
Operations for managing tenants
