thetumbled opened a new pull request, #21951:
URL: https://github.com/apache/pulsar/pull/21951
### Motivation
#21718 fix a problem that the compaction reader stuck when meet a marker
message, such as transaction marker. The reason why the reader stuck is that
`hasMessageAvailable()` rely on the `getLastMessageId`, which return the
`MessageId` `m1` of the marker message, but the server will filter marker
message transparently, the reader need to read to `m1` until the compaction
end, but the server will not dispatch the marker message(m1) to client, so the
compaction reader stuck.
The solution #21718 take is do not filter such message when the subscription
name is `__compaction`, and transfer the responsibility of filtering to
user(compaction reader), so that the `readNext` and `hasMessageAvailable` is
consistent.
But, such patch could only fix the problem of compaction, the recovery of
`TopicTransactionBuffer` also use such pattern of code, and normal users may
use such kind of logic to do something.
### Modifications
The root cause of such kind of stuck problem is that, **`getLastMessageId`
do not validate message completely,** though it has do some kind of check like
`maxReadPosition`.
We need to refactor `getLastMessageId` to ensure that message corresponding
to the id it return is valid.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads
(10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update
later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: https://github.com/thetumbled/pulsar/pull/34
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
