dragosvictor opened a new pull request, #21682:
URL: https://github.com/apache/pulsar/pull/21682
<!--
### Contribution Checklist
- PR title format should be *[type][component] summary*. For details, see
*[Guideline - Pulsar PR Naming
Convention](https://pulsar.apache.org/contribute/develop-semantic-title/)*.
- Fill out the template below to describe the changes contributed by the
pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from
multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and
this checklist, leaving only the filled out template below.
-->
PIP: [PIP-307](https://github.com/apache/pulsar/pull/20748)
### Motivation
Topic unloading can be sped up, as described by PIP-307, by forwarding the
target broker lookup data to clients (both producers and consumers). PR
https://github.com/apache/pulsar/pull/21408 added support for the producers,
this PR introduces similar functionality for the consumers.
### Modifications
When the load balancer reassigns a bundle from an old 'source' broker to a
new 'target' broker, the consumers are currently forcefully disconnected. The
expectation is that they will issue topic lookup calls to locate the new broker.
This PR forwards this information to the consumers as part of the disconnect
workflow, allowing them to connect directly to the target broker and skip the
lookups in the process.
Specifically:
- Added an optional `assignedBrokerLookupData` to the `CloseConsumer`
command. The consumer uses this information to attempt to connect to the broker
once. If the connection fails, it falls back to topic lookups.
- When the ExtensibleLoadManager is unloading a bundle, the source broker
first closes the topics _without_ disconnecting the clients. When the bundle
has been fully migrated to the target broker, the source broker closes the
topics again, this time disconnecting the clients, along with sending the
target broker information to the clients.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- Extended existing unit tests
`testTransferClientReconnectionWithoutLookup` and
`testUnloadClientReconnectionWithLookup` to cover both producer and consumer
behavior, as well as all subscription types. Ran the suite 100 times to make
sure it was stable.
- Manually verified the behavior using `pulsar-perf` in a k8s environment.
Asserted that all messages pass through during a load balancing run, while no
lookups are performed by the clients.
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update
later -->
- [X] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: (https://github.com/dragosvictor/pulsar/pull/1)
<!--
After opening this PR, the build in apache/pulsar will fail and instructions
will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since
the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed
in
a forked repository.
The tests will be run in the forked repository until all PR review comments
have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
