This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 12864e8 Compaction documentation follow-up (#1871)
12864e8 is described below
commit 12864e8a3dd95977aef406a6499dc30bfc5432c0
Author: Luc Perkins <[email protected]>
AuthorDate: Wed Jun 20 16:39:59 2018 -0700
Compaction documentation follow-up (#1871)
* clarification around compacted vs non-compacted consumers
* clarify that no-key msgs are ignored
* theoretical clarifications
* clarify language around listening/changes
---
site/docs/latest/cookbooks/compaction.md | 13 ++++++++++---
site/docs/latest/getting-started/ConceptsAndArchitecture.md | 10 ++++------
2 files changed, 14 insertions(+), 9 deletions(-)
diff --git a/site/docs/latest/cookbooks/compaction.md
b/site/docs/latest/cookbooks/compaction.md
index 3b11400..035dfd4 100644
--- a/site/docs/latest/cookbooks/compaction.md
+++ b/site/docs/latest/cookbooks/compaction.md
@@ -11,13 +11,18 @@ To use compaction:
* Compaction can be configured to run [automatically](#automatic), or you can
manually [trigger](#trigger) compaction using the Pulsar administrative API.
* Your {% popover consumers %} must be [configured](#config) to read from
compacted topics ([Java consumers](#java), for example, have a `readCompacted`
setting that must be set to `true`). If this configuration is not set,
consumers will still be able to read from the non-compacted topic.
+{% include admonition.html type="warning" content="Compaction only works on
messages that have keys (as in the stock ticker example the stock symbol serves
as the key for each message). Keys can thus be thought of as the axis along
which compaction is applied. Messages that don't have keys are simply ignored
by compaction." %}
+
## When should I use compacted topics? {#when}
-The classic example of a topic that could benefit from compaction would be a
stock ticker topic through which {% popover consumers %} can access up-to-date
values for specific stocks. On a stock ticker topic you only care about the
most recent value of each stock; "historical values" don't matter, so there's
no need to read through outdated data when processing a topic's messages.
+The classic example of a topic that could benefit from compaction would be a
stock ticker topic through which {% popover consumers %} can access up-to-date
values for specific stocks. Imagine a scneario in which messages carrying stock
value data use the stock symbol as the key (`GOOG`, `AAPL`, `TWTR`, etc.).
Compacting this topic would give consumers on the topic two options:
+
+* They can read from the "original," non-compacted topic in case they need
access to "historical" values, i.e. the entirety of the topic's messages.
+* They can read from the compacted topic if they only want to see the most
up-to-date messages.
-{% include admonition.html type="info" content="For topics where older values
are important, for example when you need to process a long series of messages
in order, many of which have the same key, compaction is unnecessary and could
possibly even be harmful." %}
+Thus, if you're using a Pulsar topic called `stock-values`, some consumers
could have access to all messages in the topic (perhaps because they're
performing some kind of number crunching of all values in the last hour) while
the consumers used to power the real-time stock ticker only see the compacted
topic (and thus aren't forced to process outdated messages). Which variant of
the topic any given consumer pulls messages from is determined by the
consumer's [configuration](#config).
-{% include admonition.html type="warning" content="Compaction only works on
topics where each message has a key (as in the stock ticker example, where the
stock symbol serves as the key). Keys can be thought of as the axis along which
compaction is applied." %}
+{% include admonition.html type="success" content="One of the benefits of
compaction in Pulsar is that you aren't forced to choose between compacted and
non-compacted topics, as the compaction process leaves the original topic as-is
and essentially adds an alternate topic. In other words, you can run compaction
on a topic and consumers that need access to the non-compacted version of the
topic will not be adversely affected." %}
## Configuring compaction to run automatically {#automatic}
@@ -48,6 +53,8 @@ $ bin/pulsar compact-topic \
--topic persistent://my-tenant-namespace/my-topic
```
+{% include admonition.html type="info" content="Running compaction in its own
process is recommended when you want to avoid interfering with the broker's
performance. Broker performance should only be affected, however, when running
compaction on topics with a large keyspace (i.e when there are many keys on the
topic). The first phase of the compaction process keeps a copy of each key in
the topic, which can create memory pressure as the number of keys grows. Using
the `pulsar-admin topi [...]
+
The `pulsar compact-topic` command communicates with
[ZooKeeper](https://zookeeper.apache.org) directly. In order to establish
communication with ZooKeeper, though, the `pulsar` CLI tool will need to have a
valid [broker configuration](../../Configuration#broker). You can either supply
a proper configuration in `conf/broker.conf` or specify a non-default location
for the configuration:
```bash
diff --git a/site/docs/latest/getting-started/ConceptsAndArchitecture.md
b/site/docs/latest/getting-started/ConceptsAndArchitecture.md
index 96c0be2..e07c5f9 100644
--- a/site/docs/latest/getting-started/ConceptsAndArchitecture.md
+++ b/site/docs/latest/getting-started/ConceptsAndArchitecture.md
@@ -551,7 +551,7 @@ For some use cases consumers don't need a complete "image"
of the topic log. The
Pulsar's topic compaction feature:
-* Allos for much more efficient "rewind" through topic logs
+* Allows for faster "rewind" through topic logs
* Applies only to [persistent topics](#persistent-storage)
* Triggered automatically when the backlog reaches a certain size or can be
triggered manually via the command line. See the [Topic compaction
cookbook](../../cookbooks/compaction)
* Is conceptually and operationally distinct from [retention and
expiry](#message-retention-and-expiry). Topic compaction *does*, however,
respect retention. If retention has removed a message from the message backlog
of a topic, the message will also not be readable from the compacted topic
ledger.
@@ -561,13 +561,11 @@ Pulsar's topic compaction feature:
### How topic compaction works
-When topic compaction is triggered [via the CLI](../../cookbooks/compaction),
Pulsar will iterate over the entire topic from beginning to end. For each key
that it encounters the {% popover broker %} responsible will keep a record of
the latest occurrence of that key. When this iterative process is finished, the
broker will create a [BookKeeper ledger](#ledgers) to store the compacted topic.
+When topic compaction is triggered [via the CLI](../../cookbooks/compaction),
Pulsar will iterate over the entire topic from beginning to end. For each key
that it encounters the compaction routine will keep a record of the latest
occurrence of that key.
-After that, the broker will make a second iteration through each message on
the topic. For each message, if the key matches the latest occurrence of that
key, then the key's data payload, message ID, and metadata will be written to
the new BookKeeper ledger (the one that was created when compaction was
manually initiated). If the key doesn't match the latest then the message will
be skipped and left alone. If any given message has an empty payload, it will
be skipped and considered delet [...]
+After that, the broker will create a new [BookKeeper ledger](#ledgers) and
make a second iteration through each message on the topic. For each message, if
the key matches the latest occurrence of that key, then the key's data payload,
message ID, and metadata will be written to the newly created ledger. If the
key doesn't match the latest then the message will be skipped and left alone.
If any given message has an empty payload, it will be skipped and considered
deleted (akin to the conc [...]
-{% include admonition.html type="info" title="Compaction leaves the original
topic intact" %}
-
-In addition to performing compaction, Pulsar {% popover brokers %} listen for
changes on each topic's metadata. If the ledger for the topic changes:
+After the initial compaction operation, the Pulsar {% popover broker %} that
owns the topic is notified whenever any future changes are made to the
compaction horizon and compacted backlog. When such changes occur:
* Clients (consumers and readers) that have read compacted enabled will
attempt to read messages from a topic and either:
* Read from the topic like normal (if the message ID is greater than or
equal to the compaction horizon) or
