Jason918 opened a new pull request #11960: URL: https://github.com/apache/pulsar/pull/11960
Implementation of [PIP-63](https://github.com/apache/pulsar/wiki/PIP-63%3A-Readonly-Topic-Ownership-Support) ### Motivation See [PIP-63](https://github.com/apache/pulsar/wiki/PIP-63%3A-Readonly-Topic-Ownership-Support) ### Modifications As described in [PIP-63](https://github.com/apache/pulsar/wiki/PIP-63%3A-Readonly-Topic-Ownership-Support). This pr comes with these key changes. 1. First of all, readeonly topic, more pricisely readonly namespace, is defined by `writerNamespace` in `org.apache.pulsar.common.policies.data.Policies` of this namespace. For example, when `writerNamespace` is set as 'NS-W' in the `Policies` of a namespace 'NS-R', we can consume all the topics from namespace 'NS-W' through namespace 'NS-R'. If a consumer client subscribes a reader topic "persistent://tenant/NS-R/topic", it's effectively subscribing the writer topic "persistent://tenant/NS-W/topic". Using namespace level readonly configuration instead of topic level configuration, we can use namespace isolation strategy to separate one brokers clusters to writer-brokers and readonly-brokers. 2. How readonly topic owner read data Related logic is mostly handled in `org.apache.bookkeeper.mledger.impl.RemoteManagedLedgerImpl` which extends `ManagedLedgerImpl` for readonly topics, and changed some key behaviors, including - Forbid all entry modification ops, like 'addEntry' and 'truncate'. - Read all ledger info from zk, and update LAC by call `org.apache.bookkeeper.client.api.ReadHandle#readLastAddConfirmedAsync` of the last ledger in the ledger list. See method `RemoteManagedLedgerImpl#asyncUpdateLastConfirmedEntry()` for details. - Using a `org.apache.pulsar.client.api.CursorClient` for managing cursors in writer topic. And it's RemoteManagedCursorImpl which extends ManagedCursorImpl used for cursor management in readonly topic. - For consumers, each time `RemoteManagedCursorImpl#checkForNewEntries` will trigger `asyncUpdateLastConfirmedEntry` to update LAC from bookies. It's not quite efficient for now, but I look forward to optimize this to a long polling way from broker (or bookie) in the following PRs, since this PR is already quite big for reviewers. 3. How does readonly topic owner keep metadata in-sync Add a method `watchManagedLedgerInfo` in `org.apache.bookkeeper.mledger.impl.MetaStore`. Every `RemoteManagedLedgerImpl` will watch its ManagedLedgerInfo to see if there is a new ledger rolled. 4. How does readonly topic owner handle acknowledgment A few cursor related RPCs is added for the acknowledgment including - GET_CURSOR - CREATE_CURSOR - DELETE_CURSOR - UPDATE_CURSOR See `pulsar-common/src/main/proto/PulsarApi.proto` for details. A CursorClient is added for readonly topics to use these apis. ### Verifying this change - [ ] Make sure that the change passes the CI checks. This change added tests and can be verified as follows: - RemoteManagedCursorImplTest.java - RemoteManagedLedgerImplTest.java - CursorClientTest.java - ReadonlyTopicOwnerTest.java ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: (no) - The wire protocol: (no) - The rest endpoints: (no) - The admin cli options: (no) - Anything that affects deployment: (no) ### Documentation Check the box below and label this PR (if you have committer privilege). Need to update docs? - [ ] doc-required (If you need help on updating docs, create a doc issue) - [ ] no-need-doc (Please explain why) - [ ] doc (If this PR contains doc changes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
