dang-stripe opened a new issue, #11001: URL: https://github.com/apache/pinot/issues/11001
we recently had some dns instability issues w/ our kafka cluster that caused consumer creation to fail since it couldn't resolve broker dns. while investigating, we noticed that we were creating N consumers every time a new realtime segment was created where N is the number of partitions on the topic. we have some topics w/ a high partition count like 200. ``` [2023-06-27 02:50:30.459797] INFO [KafkaConsumer] [HelixTaskExecutor-message_handle_thread_27:17] [Consumer clientId=example_table-example_topic-5, groupId=pinot-table-group1] Subscribed to partition(s): example_topic-100 [2023-06-27 02:50:30.459840] INFO [KafkaConsumer] [HelixTaskExecutor-message_handle_thread_27:17] [Consumer clientId=example_table-example_topic-5, groupId=pinot-table-group1] Subscribed to partition(s): example_topic-132 ... ``` it seems like the default implementation of [`computePartitionGroupMetadata`](https://github.com/apache/pinot/blob/master/pinot-spi/src/main/java/org/apache/pinot/spi/stream/StreamMetadataProvider.java#L65) creates all these consumers. i'm wondering if there are any blockers to using the `KafkaAdminClient`'s listOffsets call to achieve the same thing: https://kafka.apache.org/28/javadoc/org/apache/kafka/clients/admin/KafkaAdminClient.html#listOffsets(java.util.Map,org.apache.kafka.clients.admin.ListOffsetsOptions) here is where it's getting called for new realtime segment creation: https://github.com/apache/pinot/blob/fc26d6d8975b4cd46e26e460236a30e8b1eb2cde/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java#L1547-L1549 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
