poorbarcode opened a new pull request, #18620:
URL: https://github.com/apache/pulsar/pull/18620
### Motivation
Configuration `autoSkipNonRecoverableData` is designed to turn this feature
on if we can accept partial data loss. When a ledger is lost, the broker will
still work. But now we have this problem: If a ledger is lost, consumer and
producer can work, but the cursor mark delete position can not forward.
For example:
1. `topic-1` has 3 ledgers:
```
{
ledegr-1 : [1:1],[1:2],[1:3],[1:4],[1:5]
ledegr-2: [2:1],[2:2],[2:3],[2:4],[2:5]
ledegr-3: [3:1],[3:2],[3:3],[3:4],[3:5]
}
```
2. individual ack `[1:1], [2:5]`, then the cursor.markDeletedPosition will
be `[1:1]`, and `individualDeletedMessages` has one position `[2:5]`.
3. We've lost `ledger-2`.
4. <strong>High light</strong>Then the consumers and producers can works ok.
But the `markDeletedPosition` stay at `[1:5]`, and even calls `unload topic`
will not solve this problem
The root cause is: When an unrecoverable ledger is found, the read position
of the cursor is simply moved forward, but `individualDeletedMessages` and
`batchDeletedIndexes` are ignored.
### Modifications
- When an unrecoverable ledger is found, remove the records
in`individualDeletedMessages` and `batchDeletedIndexes`.
- When the managed cursor is recovered, check whether there are invalid
records in `individualDeletedMessages` and `batchDeletedIndexes` and print a
warning log.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. Please attach the local
preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR
description, or else your PR might not get merged. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update
later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository:
- https://github.com/poorbarcode/pulsar/pull/44
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
