coderzc commented on code in PR #25134: URL: https://github.com/apache/pulsar/pull/25134#discussion_r2734548551
########## pip/pip-452.md: ########## @@ -0,0 +1,263 @@ +# PIP-452: Customizable topic listing of namespace with properties + +# Motivation +Currently, the CommandGetTopicsOfNamespace logic in the Pulsar Broker is hard-coded to scan the metadata store (ZooKeeper) for all children nodes under a namespace. + +This implementation limits the flexibility required for complex multi-tenant scenarios: + +No Client Context: The broker cannot distinguish who is asking for the topics or why. It cannot filter topics based on client properties (these client properties may correspond to, or be derived from topic properties). + +Inefficient Filtering: For namespaces with millions of topics, the broker must fetch the full list into memory before applying the topics_pattern regex. There is no way to "push down" the filtering to the data source (e.g., a database with an index). + +To address these issues, I propose making the topic listing logic pluggable and extending the protocol to accept client properties. + +# Goals + +## In Scope + +Protocol: Add a properties field to `CommandGetTopicsOfNamespace` to carry client-side context. + +Broker: Extend `BrokerInterceptor` to allow plugins to intercept the "Get Topics" request. + +Client: Update the Java Client to forward Consumer properties to the lookup service when using Regex subscriptions. + +Admin API & CLI: Update the REST API and CLI to accept properties for listing topics in a namespace. + +Configuration: Add a broker configuration to enable/disable topic list watching. + +## Out of Scope + +Not support topic list watching for customized topic listing in this PIP. It can be considered in future work. +If you want to use this feature, you need to disable topic list watcher by set `enableBrokerTopicListWatcher = false`. + +# High Level Design +We will modify the Pulsar Protocol to carry a properties map. On the Broker side, we will add a new method to the BrokerInterceptor interface. +The default implementation will preserve the existing behavior (fetching from ZooKeeper). The Broker's connection handler will simply delegate the request to this strategy. + +# Detailed Design + +## Public-facing Changes + +### Configuration + +broker.conf +```properties +# Enables watching topic add/remove events on broker side. It is separated from enableBrokerSideSubscriptionPatternEvaluation. +enableBrokerTopicListWatcher = true Review Comment: > ## Out of Scope >Not support topic list watching for customized topic listing in this PIP. It can be considered in future work. >If you want to use this feature, you need to disable topic list watcher by set `enableBrokerTopicListWatcher = false`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
