ivankelly commented on a change in pull request #1466: Topic compaction
documentation
URL: https://github.com/apache/incubator-pulsar/pull/1466#discussion_r191705439
##########
File path: site/docs/latest/cookbooks/compaction.md
##########
@@ -0,0 +1,107 @@
+---
+title: Topic compaction cookbook
+tags: [admin, clients, compaction]
+---
+
+Pulsar's [topic
compaction](../../getting-started/ConceptsAndArchitecture#compaction) feature
enables you to create **compacted** topics in which older, "obscured" entries
are pruned from the topic, allowing for faster reads through the topic's
history (which messages are deemed obscured/outdated/irrelevant will depend on
your use case).
+
+To use compaction:
+
+* You need to give messages keys, as topic compaction in Pulsar takes place on
a *per-key basis* (i.e. messages are compacted based on their key). For a stock
ticker use case, the stock symbol---e.g. `AAPL` or `GOOG`---could serve as the
key (more on this [below](#when)). Messages without keys will be left alone by
the compaction process.
+* You must manually [trigger](#trigger) compaction using the Pulsar
administrative API. This will both run a compaction operation *and* mark the
topic as a compacted topic.
+* Your {% popover consumers %} must be [configured](#config) to read from
compacted topics ([Java consumers](#java), for example, have a `readCompacted`
setting that must be set to `true`). If this configuration is not set,
consumers will still be able to read from the non-compacted topic.
+
+## When should I use compacted topics? {#when}
+
+The classic example of a topic that could benefit from compaction would be a
stock ticker topic through which {% popover consumers %} can access up-to-date
values for specific stocks. On a stock ticker topic you only care about the
most recent value of each stock; "historical values" don't matter, so there's
no need to read through outdated data when processing a topic's messages.
+
+{% include admonition.html type="info" content="For topics where older values
are important, for example when you need to process a long series of messages
in order, many of which have the same key, compaction is unnecessary and could
possibly even be harmful." %}
+
+{% include admonition.html type="warning" content="Compaction only works on
topics where each message has a key (as in the stock ticker example, where the
stock symbol serves as the key). Keys can be thought of as the axis along which
compaction is applied." %}
+
+## Triggering compaction {#trigger}
+
+In order to run compaction on a topic, you need to use the [`topics
compact`](../../CliTools#pulsar-admin-topics-compact) command for the
[`pulsar-admin`](../../CliTools#pulsar-admin) CLI tool. Here's an example:
+
+```bash
+$ bin/pulsar-admin topics compact \
+ persistent://my-tenant/my-namespace/my-topic
+```
+
+The `pulsar-admin` tool runs compaction via the Pulsar [REST
API](../../reference/RestApi). To run compaction in its own dedicated process,
i.e. *not* through the REST API, you can use the [`pulsar
compact-topic`](../../CliTools#pulsar-compact-topic) command. Here's an example:
Review comment:
We should give some guidance on why one would want to run compaction in its
own process. Generally it's to avoid interfering with the broker's performance,
which will only be the case where the keyspace is large. When compaction is
run, the first phase keeps a copy of each key in the topic. This could create
memory pressure on the broker if run via rest api.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
