sijie closed pull request #1871: Compaction documentation follow-up
URL: https://github.com/apache/incubator-pulsar/pull/1871
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/site/docs/latest/cookbooks/compaction.md
b/site/docs/latest/cookbooks/compaction.md
index 3b11400ef8..035dfd4ff0 100644
--- a/site/docs/latest/cookbooks/compaction.md
+++ b/site/docs/latest/cookbooks/compaction.md
@@ -11,13 +11,18 @@ To use compaction:
* Compaction can be configured to run [automatically](#automatic), or you can
manually [trigger](#trigger) compaction using the Pulsar administrative API.
* Your {% popover consumers %} must be [configured](#config) to read from
compacted topics ([Java consumers](#java), for example, have a `readCompacted`
setting that must be set to `true`). If this configuration is not set,
consumers will still be able to read from the non-compacted topic.
+{% include admonition.html type="warning" content="Compaction only works on
messages that have keys (as in the stock ticker example the stock symbol serves
as the key for each message). Keys can thus be thought of as the axis along
which compaction is applied. Messages that don't have keys are simply ignored
by compaction." %}
+
## When should I use compacted topics? {#when}
-The classic example of a topic that could benefit from compaction would be a
stock ticker topic through which {% popover consumers %} can access up-to-date
values for specific stocks. On a stock ticker topic you only care about the
most recent value of each stock; "historical values" don't matter, so there's
no need to read through outdated data when processing a topic's messages.
+The classic example of a topic that could benefit from compaction would be a
stock ticker topic through which {% popover consumers %} can access up-to-date
values for specific stocks. Imagine a scneario in which messages carrying stock
value data use the stock symbol as the key (`GOOG`, `AAPL`, `TWTR`, etc.).
Compacting this topic would give consumers on the topic two options:
+
+* They can read from the "original," non-compacted topic in case they need
access to "historical" values, i.e. the entirety of the topic's messages.
+* They can read from the compacted topic if they only want to see the most
up-to-date messages.
-{% include admonition.html type="info" content="For topics where older values
are important, for example when you need to process a long series of messages
in order, many of which have the same key, compaction is unnecessary and could
possibly even be harmful." %}
+Thus, if you're using a Pulsar topic called `stock-values`, some consumers
could have access to all messages in the topic (perhaps because they're
performing some kind of number crunching of all values in the last hour) while
the consumers used to power the real-time stock ticker only see the compacted
topic (and thus aren't forced to process outdated messages). Which variant of
the topic any given consumer pulls messages from is determined by the
consumer's [configuration](#config).
-{% include admonition.html type="warning" content="Compaction only works on
topics where each message has a key (as in the stock ticker example, where the
stock symbol serves as the key). Keys can be thought of as the axis along which
compaction is applied." %}
+{% include admonition.html type="success" content="One of the benefits of
compaction in Pulsar is that you aren't forced to choose between compacted and
non-compacted topics, as the compaction process leaves the original topic as-is
and essentially adds an alternate topic. In other words, you can run compaction
on a topic and consumers that need access to the non-compacted version of the
topic will not be adversely affected." %}
## Configuring compaction to run automatically {#automatic}
@@ -48,6 +53,8 @@ $ bin/pulsar compact-topic \
--topic persistent://my-tenant-namespace/my-topic
```
+{% include admonition.html type="info" content="Running compaction in its own
process is recommended when you want to avoid interfering with the broker's
performance. Broker performance should only be affected, however, when running
compaction on topics with a large keyspace (i.e when there are many keys on the
topic). The first phase of the compaction process keeps a copy of each key in
the topic, which can create memory pressure as the number of keys grows. Using
the `pulsar-admin topics compact` command to run compaction through the REST
API should present no issues in the overwhelming majority of cases; using
`pulsar compact-topic` should correspondingly be considered an edge case." %}
+
The `pulsar compact-topic` command communicates with
[ZooKeeper](https://zookeeper.apache.org) directly. In order to establish
communication with ZooKeeper, though, the `pulsar` CLI tool will need to have a
valid [broker configuration](../../Configuration#broker). You can either supply
a proper configuration in `conf/broker.conf` or specify a non-default location
for the configuration:
```bash
diff --git a/site/docs/latest/getting-started/ConceptsAndArchitecture.md
b/site/docs/latest/getting-started/ConceptsAndArchitecture.md
index d6efb4cd5f..d439ff9b45 100644
--- a/site/docs/latest/getting-started/ConceptsAndArchitecture.md
+++ b/site/docs/latest/getting-started/ConceptsAndArchitecture.md
@@ -551,7 +551,7 @@ For some use cases consumers don't need a complete "image"
of the topic log. The
Pulsar's topic compaction feature:
-* Allos for much more efficient "rewind" through topic logs
+* Allows for faster "rewind" through topic logs
* Applies only to [persistent topics](#persistent-storage)
* Triggered automatically when the backlog reaches a certain size or can be
triggered manually via the command line. See the [Topic compaction
cookbook](../../cookbooks/compaction)
* Is conceptually and operationally distinct from [retention and
expiry](#message-retention-and-expiry). Topic compaction *does*, however,
respect retention. If retention has removed a message from the message backlog
of a topic, the message will also not be readable from the compacted topic
ledger.
@@ -561,13 +561,11 @@ Pulsar's topic compaction feature:
### How topic compaction works
-When topic compaction is triggered [via the CLI](../../cookbooks/compaction),
Pulsar will iterate over the entire topic from beginning to end. For each key
that it encounters the {% popover broker %} responsible will keep a record of
the latest occurrence of that key. When this iterative process is finished, the
broker will create a [BookKeeper ledger](#ledgers) to store the compacted topic.
+When topic compaction is triggered [via the CLI](../../cookbooks/compaction),
Pulsar will iterate over the entire topic from beginning to end. For each key
that it encounters the compaction routine will keep a record of the latest
occurrence of that key.
-After that, the broker will make a second iteration through each message on
the topic. For each message, if the key matches the latest occurrence of that
key, then the key's data payload, message ID, and metadata will be written to
the new BookKeeper ledger (the one that was created when compaction was
manually initiated). If the key doesn't match the latest then the message will
be skipped and left alone. If any given message has an empty payload, it will
be skipped and considered deleted (akin to the concept of
[tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value
databases). At the end of this second iteration through the topic, the newly
created BookKeeper ledger is closed and two things are written to the topic's
metadata: the ID of the BookKeeper ledger and the message ID of the last
compacted message (this is known as the **compaction horizon** of the topic).
Once this metadata is written compaction is complete.
+After that, the broker will create a new [BookKeeper ledger](#ledgers) and
make a second iteration through each message on the topic. For each message, if
the key matches the latest occurrence of that key, then the key's data payload,
message ID, and metadata will be written to the newly created ledger. If the
key doesn't match the latest then the message will be skipped and left alone.
If any given message has an empty payload, it will be skipped and considered
deleted (akin to the concept of
[tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value
databases). At the end of this second iteration through the topic, the newly
created BookKeeper ledger is closed and two things are written to the topic's
metadata: the ID of the BookKeeper ledger and the message ID of the last
compacted message (this is known as the **compaction horizon** of the topic).
Once this metadata is written compaction is complete.
-{% include admonition.html type="info" title="Compaction leaves the original
topic intact" %}
-
-In addition to performing compaction, Pulsar {% popover brokers %} listen for
changes on each topic's metadata. If the ledger for the topic changes:
+After the initial compaction operation, the Pulsar {% popover broker %} that
owns the topic is notified whenever any future changes are made to the
compaction horizon and compacted backlog. When such changes occur:
* Clients (consumers and readers) that have read compacted enabled will
attempt to read messages from a topic and either:
* Read from the topic like normal (if the message ID is greater than or
equal to the compaction horizon) or
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
