This is an automated email from the ASF dual-hosted git repository.
baodi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 2db2e76d10c [improve][PIP] PIP-353: Improve transaction message
visibility for peek-messages (#22746)
2db2e76d10c is described below
commit 2db2e76d10c8b50e73ee11a0e081579c8738d64b
Author: Baodi Shi <[email protected]>
AuthorDate: Fri May 24 16:57:51 2024 +0800
[improve][PIP] PIP-353: Improve transaction message visibility for
peek-messages (#22746)
---
pip/pip-353.md | 134 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 134 insertions(+)
diff --git a/pip/pip-353.md b/pip/pip-353.md
new file mode 100644
index 00000000000..4315bdab0eb
--- /dev/null
+++ b/pip/pip-353.md
@@ -0,0 +1,134 @@
+# PIP-353: Improve transaction message visibility for peek-messages cli
+
+## Background knowledge
+
+This PIP addresses enhancements message visibility for the peek-message CLI,
specifically related to transaction messages and markers.
+Currently, when peeking messages, users may encounter `server internal marker`
and `uncommitted` or `aborted` for transaction message,
+which should typically be hidden to ensure data integrity and consistency.
+
+### Transaction Markers
+Transaction markers are internal messages used by Pulsar to denote the start,
end, and status (commit/abort) of a transaction. They are not meant to be
consumed by clients.
+
+Similarly, other [internal
markers](https://github.com/apache/pulsar/blob/ed5d94ccfdf4eba77678454945a2c3719dce2268/pulsar-common/src/main/proto/PulsarMarkers.proto#L25-L38)
+used by Pulsar for various system operations should also not be visible to
clients as they could lead to confusion and misinterpretation of the data.
+
+### Transaction Messages
+- Uncommitted Messages: These should not be visible to consumers until the
transaction is committed.
+- Aborted Messages: These should be filtered out and never made visible to
consumers.
+
+## Motivation
+
+The current implementation exposes all messages, including transaction markers
and messages from uncommitted or aborted transactions, when peeking.
+This behavior can confuse users and lead to incorrect data handling. The
proposal aims to provide more control over what types of messages are visible
during message peeking operations.
+
+## Goals
+
+### In Scope
+
+- Implement flags to selectively display `server markers`, `uncommitted
messages`, and `aborted messages` in peek operations.
+- Set the default behavior to only show messages from committed transactions
to ensure data integrity.
+
+### Out of Scope
+- Any modifications to the core transaction handling mechanism.
+- Any changes to consumer logic.
+
+## High Level Design
+
+The proposal introduces three new flags to the `peek-messages` command:
+
+1. `--show-server-marker`: Controls the visibility of server markers (default:
`false`).
+2. `--show-txn-uncommitted`: Controls the visibility of messages from
uncommitted transactions (default: `false`).
+3. `--show-txn-aborted`: Controls the visibility of messages from aborted
transactions (default: `false`).
+
+These flags will allow administrators and developers to tailor the peek
functionality to their needs, improving the usability and security of message
handling in transactional contexts.
+
+## Detailed Design
+
+### Design & Implementation Details
+
+To support the `--show-server-marker` and `--show-txn-aborted`,
`--show-txn-uncommitted` flags, needs to introduce specific tag into the
`headers` of messages returned by the
+[peekNthMessage REST
API](https://github.com/apache/pulsar/blob/8ca01cd42edfd4efd986f752f6f8538ea5bf4f94/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java#L1892-L1905).
+
+- `X-Pulsar-marker-type`: Already exists.
+- `X-Pulsar-txn-uncommitted`: This entry is determined to be an uncommitted
transaction by comparing its `position` and `maxReadPosition`.
+- `X-Pulsar-txn-aborted`: It is determined to be aborted by calling the
`persistentTopic.isAbort()` method.
+
+Then, In the CLI, these markers can be used to determine whether to filter out
these messages and proceed to the next one. For an implementation example,
+see the following code:
[https://github.com/shibd/pulsar/pull/34](https://github.com/shibd/pulsar/pull/34)
+
+### Public-facing Changes
+
+#### CLI Command Flags
+
+New command line flags added for the `bin/pulsar-admin topics peek-messages`
command:
+
+| Flag | Abbreviation | Type | Default | Description
|
+|--------------------------|--------------|---------|---------|----------------------------------------------------------------|
+| `--show-server-marker` | `-ssm` | Boolean | `false` | Enables the
display of internal server write markers. |
+| `--show-txn-uncommitted` | `-stu` | Boolean | `false` | Enables the
display of messages from uncommitted transactions. |
+| `--show-txn-aborted` | `-sta` | Boolean | `false` | Enables the
display of messages from aborted transactions. |
+
+
+## Public-facing Changes
+
+Add two methods to the admin.Topics() interface.
+
+```java
+ /**
+ * Peek messages from a topic subscription.
+ *
+ * @param topic
+ * topic name
+ * @param subName
+ * Subscription name
+ * @param numMessages
+ * Number of messages
+ * @param showServerMarker
+ * Enables the display of internal server write markers
+ * @param showTxnAborted
+ * Enables the display of messages from aborted transactions
+ * @param showTxnUncommitted
+ * Enables the display of messages from uncommitted transactions
+ * @return
+ * @throws NotAuthorizedException
+ * Don't have admin permission
+ * @throws NotFoundException
+ * Topic or subscription does not exist
+ * @throws PulsarAdminException
+ * Unexpected error
+ */
+ List<Message<byte[]>> peekMessages(String topic, String subName, int
numMessages,
+ boolean showServerMarker, boolean
showTxnAborted,
+ boolean showTxnUncommitted) throws
PulsarAdminException;
+
+ /**
+ * Peek messages from a topic subscription asynchronously.
+ *
+ * @param topic
+ * topic name
+ * @param subName
+ * Subscription name
+ * @param numMessages
+ * Number of messages
+ * @param showServerMarker
+ * Enables the display of internal server write markers
+ * @param showTxnAborted
+ * Enables the display of messages from aborted transactions
+ * @param showTxnUncommitted
+ * Enables the display of messages from uncommitted transactions
+ * @return a future that can be used to track when the messages are
returned
+ */
+ CompletableFuture<List<Message<byte[]>>> peekMessagesAsync(String topic,
String subName, int numMessages,
+ boolean
showServerMarker, boolean showTxnAborted,
+ boolean
showTxnUncommitted);
+```
+
+## Backward & Forward Compatibility
+
+### Revert
+Reverting to a previous version of Pulsar without this feature will remove the
additional flags from the CLI. Users who prefer the new behavior will need to
manually adjust their usage patterns when reverting.
+
+### Upgrade
+While upgrading to the new version of Pulsar that includes these changes, the
default behavior of the `peek-messages` command will change.
+Existing scripts or commands that rely on the old behavior (where transaction
markers and messages from uncommitted or aborted transactions are visible) will
need to explicitly set the new flags (`--show-server-marker`,
`--show-txn-uncommitted`, `--show-txn-aborted`) to `true` to maintain the old
behavior.
+This change is necessary as the previous default behavior did not align with
typical expectations around data visibility and integrity in transactional
systems.
\ No newline at end of file
