poorbarcode opened a new pull request, #21063:
URL: https://github.com/apache/pulsar/pull/21063
### Motivation
If we set `allowAutoTopicCreationType` to `PARTITIONED`, the flow of the
create topic progress is like the below:
1. `Client-side`: Lookup topic to get partitioned topic metadata to create a
producer.
1. `Broker-side`: Create partitioned topic metadata.
1. `Broker-side`: response `{"partitions":3}`.
1. `Client-side`: Create separate connections for each partition of the
topic.
1. `Broker-side`: Receive 3 connect requests and create 3 partition-topics.
In the `step 2` above, the flow of the progress is like the below:
1. Check the policy of topic auto-creation( the policy is
`{allowAutoTopicCreationType=PARTITIONED, defaultNumPartitions=3}` )
1. Check the partitioned topic metadata already exists.
1. Try to create the partitioned topic metadata if it does not exist.
1. If created failed by the partitioned topic metadata already exists( maybe
another broker is also creating now), read partitioned topic metadata from the
metadata store and respond to the client.
There is a race condition that makes the client get non-partitioned metadata
of the topic:
| time | `broker-1` | `broker-2` |
| --- | --- | --- |
| 1 | get policy: `PARTITIONED, 3` | get policy: `PARTITIONED, 3` |
| 2 | check the partitioned topic metadata already exists | Check the
partitioned topic metadata already exists |
| 3 | Partitioned topic metadata does not exist, the metadata cache will
cache an empty optional for the path | Partitioned topic metadata does not
exist, the metadata cache will cache an empty optional for the path |
| 4 | | succeed create the partitioned topic metadata |
| 5 | Receive a ZK node changed event to invalidate the cache of the
partitioned topic metadata |
| 6 | Creating the metadata failed due to it already exists |
| 7 | Read the partitioned topic metadata again |
If `step-5` is executed later than `step-7`, `broker-1` will get an empty
optional from the cache of the partitioned topic metadata and respond
non-partitioned metadata to the client.
**What thing would make the `step-5` is executed later than `step-7`?**
Such as the issue that the PR https://github.com/apache/pulsar/pull/20303
fixed, it makes `zk operation` and `zk node changed notifications` executed in
different threads: `main-thread of ZK client` and `metadata store thread`.
Therefore, the mechanism of the lookup partitioned topic metadata is fragile
and we need to optimize it.
### Modifications
Before reading the partitioned topic metadata again, refresh the cache first.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update
later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
