codelipenghui opened a new pull request #12161:
URL: https://github.com/apache/pulsar/pull/12161
Currently, for the get last message ID request the broker returns -1:-1 if
all the original data been deleted.
```
09:51:12.156 [bookkeeper-ml-scheduler-OrderedScheduler-3-0] INFO
org.apache.pulsar.broker.service.ServerCnx - [/172.16.124.36:44443] Created
subscription on topic xxx
09:51:12.156 [bookkeeper-ml-scheduler-OrderedScheduler-3-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Reset
cursor:ManagedCursorImpl{ledger=xxx, name=__compaction, ackPos=44946:0,
readPos=44946:1} to 66425:-1 since ledger consumed completely
09:51:12.156 [BookKeeperClientWorker-OrderedExecutor-3-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [xxx] Ledger 44946
contains the current last confirmed entry 44946:0, and it is going to be deleted
09:51:12.159 [bookkeeper-ml-scheduler-OrderedScheduler-3-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [xxx] End
TrimConsumedLedgers. ledgers=1 totalSize=0
09:51:12.159 [bookkeeper-ml-scheduler-OrderedScheduler-3-0] INFO
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [xxx] Removing ledger
44946 - size: 3999
```
After the rollover task, the topic internal stats will be:
```
{
"entriesAddedCounter": 0,
"numberOfEntries": 0,
"totalSize": 0,
"currentLedgerEntries": 0,
"currentLedgerSize": 0,
"lastLedgerCreatedTimestamp": "2021-09-20T09:51:12.15Z",
"waitingCursorsCount": 29,
"pendingAddEntriesCount": 0,
"lastConfirmedEntry": "44946:0",
"state": "LedgerOpened",
"ledgers": [
{
"ledgerId": 66425,
"entries": 0,
"size": 0,
"offloaded": false,
"underReplicated": false
}
],
"cursors": {
"__compaction": {
"markDeletePosition": "44946:0",
"readPosition": "44946:1",
"waitingReadOp": false,
"pendingReadOps": 0,
"messagesConsumedCounter": 0,
"cursorLedger": -1,
"cursorLedgerLastEntry": -1,
"individuallyDeletedMessages": "[]",
"lastLedgerSwitchTimestamp": "2021-09-20T09:51:12.154Z",
"state": "NoLedger",
"numberOfEntriesSinceFirstNotAckedMessage": 1,
"totalNonContiguousDeletedMessagesRange": 0,
"subscriptionHavePendingRead": false,
"subscriptionHavePendingReplayRead": false,
"properties": {
"CompactedTopicLedger": 64365
}
}
},
"schemaLedgers": [],
"compactedLedger": {
"ledgerId": 64365,
"entries": 1,
"size": 4024,
"offloaded": false,
"underReplicated": false
}
}
```
At this time, when a reader call hasMessageAvailable(), the client will get
the last message id from the broker, the NonRecoverableLedgerException will
throw at the broker side due the ledger 44946 has been deleted.
```
12:41:40.937 [pulsar-io-4-5] INFO org.apache.pulsar.broker.service.ServerCnx
- [/172.16.124.36:53488] Created subscription on topic xxx / yyy
12:41:41.131 [BookKeeperClientWorker-OrderedExecutor-3-0] ERROR
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [xxx] Error opening
ledger for reading at position 44946:0 -
org.apache.bookkeeper.mledger.ManagedLedgerException$NonRecoverableLedgerException:
No such ledger exists on Metadata Server
```
The problem is we are not checking if there is compacted data for the topic.
If the topic has compacted data but encounter the above situation, we should
return the last message ID of the compacted Ledger to the client.
Added the test for the new changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
