This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.wiki.git
The following commit(s) were added to refs/heads/master by this push:
new 2ac7305 Add PIP-54
2ac7305 is described below
commit 2ac73050f5ee2d84b39d49d409c69dcea63b29a8
Author: lipenghui <[email protected]>
AuthorDate: Fri Jan 17 21:56:26 2020 +0800
Add PIP-54
---
...owledgement-for-local-index-of-batch-message.md | 76 ++++++++++++++++++++++
1 file changed, 76 insertions(+)
diff --git a/PIP-54:-Acknowledgement-for-local-index-of-batch-message.md
b/PIP-54:-Acknowledgement-for-local-index-of-batch-message.md
new file mode 100644
index 0000000..1b35fdb
--- /dev/null
+++ b/PIP-54:-Acknowledgement-for-local-index-of-batch-message.md
@@ -0,0 +1,76 @@
+* **Status**: Proposal
+* **Author**: Penghui Li
+* **Pull Request**: https://github.com/apache/pulsar/pull/6052
+* **Mailing List discussion**:
+* **Release**:
+
+
+## Motivation
+
+Currently, the managed cursor maintain the acked messages by mark delete
position and individual delete messages. All of the mark delete position and
individual delete messages are faced to batch messages(ledgerId,entryId). For
users, the single message is the publishing and consuming unit. Since the
managed cursor can’t maintain the local index of a batch message, broker will
dispatch acked messages(not batch) to consumers like issue
https://github.com/apache/pulsar/issues/5969.
+
+So this PIP is to support the ability to track the ack status of each batch
index to avoid dispatch acked messages to users.
+
+## Approach
+
+This approach requires the cooperation of the client and server. When the
broker dispatch messages, it will carry the batch index that has been acked.
The client will filter out the batch index that has been acked.
+
+Client need to send the batch index ack information to broker, so that broker
can maintain the batch index ack status.
+
+The managed cursor maintain the batch index ack status in memory by using a
BitSet and the BitSet can be persisted to ledger and metastore, so that can
avoid broker crash. When broker receive the batch index ack request, the acked
batch index will be add the BitSet. When broker dispatch messages to client
will get the batch message index ack status from the managed cursor and send it
to client. When all indexes of the batch message are acked, the cursor will
delete the batch message.
+
+Be careful when calculating consumer permits because client filter out the
acked batch index, so the broker need to increase the available permits equals
to acked batch indexes. Otherwise, the broker will stop dispatch messages to
that consumer because it does not have enough available permits.
+
+For save resources, transfer the acked batch indexes ranges between client and
broker. And in managed cursor also persist batch indexes ranges to the cursor
ledger and metadata store.
+
+
+## Changes
+
+### Wire protocol
+
+```proto
+message BatchMessageIndexesAckData {
+ required MessageIdData message_id = 1;
+ required int32 batchSize = 2;
+ repeated IntRange ack_indexes = 3;
+}
+```
+
+```proto
+
+message CommandMessage {
+ repeated IntRange acked_indexes = 4;
+}
+```
+
+```proto
+
+message CommandAck {
+ repeated BatchMessageIndexesAckData batch_message_ack_indexes = 8;
+}
+```
+
+### Managed ledger data formats
+
+```proto
+message BatchDeletedIndexRange {
+ required uint32 lowerIndex = 1;
+ required uint32 upperIndex = 2;
+}
+
+message BatchDeletedIndexInfo {
+ required NestedPositionInfo position = 1;
+ repeated BatchDeletedIndexRange batchDeletedIndexes = 2;
+}
+```
+
+```proto
+
+message PositionInfo {
+ // Store which index in the batch message has been deleted
+ repeated BatchDeletedIndexInfo batchDeletedIndexes = 5;
+}
+```
+
+## Configuration
+Added a flag named `batchIndexAcknowledgeEnable` in broker.conf to enable or
disable the batch index acknowledgement
\ No newline at end of file
