heesung-sn opened a new pull request, #18195:
URL: https://github.com/apache/pulsar/pull/18195
<!--
### Contribution Checklist
- PR title format should be *[type][component] summary*. For details, see
*[Guideline - Pulsar PR Naming
Convention](https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit#heading=h.trs9rsex3xom)*.
- Fill out the template below to describe the changes contributed by the
pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from
multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and
this checklist, leaving only the filled out template below.
-->
<!-- or this PR is one task of an issue -->
Master Issue: https://github.com/apache/pulsar/issues/18099
### Motivation
<!-- Explain here the context, and why you're making that change. What is
the problem you're trying to solve. -->
This PR supports `TopicCompactionStrategy` in `StrategicTwoPhaseCompactor`
and `TableView` for the internal system topics, as proposed in PIP-215.
This PR does not expose StrategicTwoPhaseCompactor for customer topics yet.
We want to expose this feature to users after proven to be stable on the new
system topic in the new broker load balancer(in PIP-192) first.
### Modifications
This PR adds the following classes to implement `StrategicTwoPhaseCompactor.
- Added `StrategicTwoPhaseCompactor` that extends `TwoPhaseCompactor`.
- Added `TopicCompactionStrategy` interface.
- Added `TopicCompactionStrategy` logic in `StrategicTwoPhaseCompactor` and
`TableViewImpl`.
- Added `CompactionReaderImpl` to scan topic messages in
StrategicTwoPhaseCompactor. It cumulatively acknowledges the read messages at
the end of StrategicTwoPhaseCompactor.
- Added `SubscriptionMode` and `SubscriptionInitialPosition` parameters in
`ReaderConfigurationData` since the `CompactionReaderImpl` reader's
subscription needs to be durable and needs to read from the earliest position.
- Added `RawBatchMessageContainerImpl` to batch and serialize messages in
`StrategicTwoPhaseCompactor`.
This PR updates the `TableViewImpl` to use `TopicCompactionStrategy` in its
data K,V map update logic.
- Added the `topicCompactionStrategy` member variable in
`TableViewConfigurationData`
- Added the `listen()` interface in `TableView` to provide an option to call
the listener actions only for the tail messages.
- Updated `TableViewImpl` and its `handleMessage()` function to consider
`TopicCompactionStrategy` when updating the data K,V map.
This PR updated the compaction test classes to reuse the test cases.
This PR updated the modifiers of the parent classes of the added classes to
access the member variables and member functions.
<!-- Describe the modifications you've done. -->
### Verifying this change
- [x] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- Extended the compaction tests to cover the same logic passes with the
`StrategicTwoPhaseCompactor`.
- Added other unit tests for the added classes.
### Does this pull request potentially affect one of the following parts:
*If the box was checked, please highlight the changes*
- [] Dependencies (add or upgrade a dependency)
- [x] The public API (table-view.listen() addition and reader and table-view
configuration update)
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. Please attach the local
preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR
description, or else your PR might not get merged. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update
later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
This PR does not enable strategic compaction for the customer topics. We
will update the doc when enabling this compaction for the customer topics.
### Matching PR in forked repository
PR in forked repository: https://github.com/heesung-sn/pulsar/pull/12
<!--
After opening this PR, the build in apache/pulsar will fail and instructions
will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since
the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed
in
a forked repository.
The tests will be run in the forked repository until all PR review comments
have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
