f1amingo opened a new issue, #10929:
URL: https://github.com/apache/rocketmq/issues/10929
### Before Creating the Enhancement Request
- [x] I have confirmed this should be classified as an enhancement.
### Summary
Several lite-topic hot paths iterate the **entire consume-queue table** and
filter by parent topic on every invocation, so their cost grows with the
*total* number of lmqs on the broker instead of the number of lmqs under the
target parent topic:
- `LiteEventDispatcher.doFullDispatchForWildcardGroup` — full CQ-table scan
per wildcard full dispatch
- `AbstractLiteLifecycleManager.getLiteTopicCount` / `collectByParentTopic`
/ `cleanByParentTopic` — full scan to collect lmqs of one parent topic
In large deployments (e.g. millions of lmqs under a single parent topic),
each wildcard dispatch or cleanup round pays an O(total lmqs) scan, causing
visible dispatch latency and CPU waste.
### Motivation
Wildcard groups are dispatched periodically and on demand; today every round
re-scans the whole CQ table even though only a small subtree (one parent topic)
is relevant. With a prefix-ordered index over lmq names, these paths can locate
the target subtree directly, turning O(total lmqs) into O(matched lmqs).
### Describe the Solution You'd Like
1. Add `LmqPrefixIndex`, an in-memory prefix-ordered index over lmq names in
`AbstractLiteLifecycleManager`:
- populated once at startup via `bootstrapLmqPrefixIndex()`
- maintained on the lmq lifecycle hot path via `onLmqCreate` /
`onLmqDelete` hooks
2. Add `forEachLiteTopicByPrefix` / `forEachLiteTopicByParent` iteration
APIs backed by the index; rewrite wildcard full dispatch, `getLiteTopicCount`,
`collectByParentTopic` and `cleanByParentTopic` on top of them (cleanup
switches to collect-then-delete to avoid lock nesting).
3. Refine the lite subscription model along the way: replace
`SubscriberWrapper` with a plain `Map<String, List<ClientGroup>>` return from
`getAllSubscribers`, simplify `getWildcardGroupClients` signature, rename
`getLiteTopicSet` to `getLmqSet`, and make `LiteCtlListener` callbacks default
methods.
No protocol or external API change; behavior stays equivalent to today.
### Describe Alternatives You've Considered
- Keep the full scan with early-break filtering (current behavior) — still
O(total lmqs) per round, unacceptable at million-scale lmq counts.
- A persistent/store-level index — much heavier and unnecessary since the
lmq set is already recoverable from the CQ table at startup.
### Additional Context
Unit tests cover the index (prefix lookup, bootstrap, create/delete hooks),
the rewritten dispatch path and the subscription model refactoring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
