suvodeep-pyne opened a new pull request, #14654: URL: https://github.com/apache/pinot/pull/14654
Introduces a new listTopics API to the StreamMetadataProvider interface, enabling the ability to enumerate topics for streaming connectors in Apache Pinot. Key Changes 1. New API in StreamMetadataProvider: - Added listTopics(Duration timeout) method to fetch the list of topics within a specified timeout period. - Introduced TopicMetadata as a representation of topic metadata, currently including the topic name but extensible for future metadata fields. 2. Implementation for Kafka Connectors (2.0 & 3.0) - Implemented the listTopics method in KafkaStreamMetadataProvider for both Kafka 2.0 and Kafka 3.0 plugins. - Utilized Kafka’s Consumer#listTopics() API to retrieve the topic list and mapped the results to the TopicMetadata structure. 3. Unit Test Coverage: - Added unit test in KafkaPartitionLevelConsumerTest to validate the listTopics functionality. 4. Backward Compatibility: - Default implementation of listTopics in StreamMetadataProvider throws UnsupportedOperationException to maintain backward compatibility with other connectors that do not override the method. Future Work: - Extend TopicMetadata to include additional metadata like partition count, replication factor, or topic configurations. - Add similar support for other streaming connectors in Apache Pinot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
