navina commented on issue #11001: URL: https://github.com/apache/pinot/issues/11001#issuecomment-1613195924
@dang-stripe The number of consumers created in a single server is same as the number of partitions being consumed on the server. Additionally, it creates multiple short-lived consumer instances to fetch topic metadata, which is what you see in `_partitionMetadataProvider` . This is purely due to how the code is organized in the kafka consumer plugin in Pinot (`KafkaStreamMetadataProvider` inherits from `KafkaPartitionLevelConnectionHandler`). I also mention it in this issue - https://github.com/apache/pinot/issues/10014#issue-1505401315 (see description Note) > i'm wondering if there are any blockers to using the KafkaAdminClient's listOffsets call to achieve the same thing: I don't think there are any. `KafkaAdminClient` didn't exist or was not stable when Pinot's kafka consumer was written. We should replace it avoid all the short-lived consumers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
