[
https://issues.apache.org/jira/browse/KAFKA-13612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482115#comment-17482115
]
Dominic Evans edited comment on KAFKA-13612 at 1/25/22, 9:26 PM:
-----------------------------------------------------------------
[~hachikuji] I have no affiliation with Burrow, but I can wager a guess as to
why they choose to read from consumer offsets directly. If you're writing any
software to monitor the lag of Kafka Consumer groups, it's much more efficient
to be asynchronously driven by the changes to the consumer offsets partitions
than it is to repeatedly poll the Kafka API, cache results and manage evictions
yourself.
Option 1: read from consumer offsets and parse the data format yourself
Option 2:
* poll every broker in the cluster with ListGroupsRequests
* send FindCoordinator requests for every group and put them in batches
* send DescribeGroupRequest to each coordinator with the batches of consumer
groups
* read the results out of the DescribeGroupResponses and go round the loop
again
was (Author: dnwe):
[~hachikuji] I have no affiliation with Burrow, but I can wager a guess as to
why they choose to read from _consumer_offsets directly. If you're writing any
software to monitor the lag of Kafka Consumer groups, it's much more efficient
to be asynchronously driven by the changes to the {_}`{_}_consumer_offsets`
partitions than it is to repeatedly poll the Kafka API, cache results and
manage evictions yourself.
Option 1: read from __consumer_offsets and parse the data format yourself
Option 2:
* poll every broker in the cluster with ListGroupsRequests
* send FindCoordinator requests for every group and put them in batches
* send DescribeGroupRequest to each coordinator with the batches of consumer
groups
* read the results out of the DescribeGroupResponses and go round the loop
again
> internal topics won't be created in metadataRequest when
> auto.create.topics.enable=false
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-13612
> URL: https://issues.apache.org/jira/browse/KAFKA-13612
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 2.8.0, 3.0.0
> Reporter: Luke Chen
> Assignee: Luke Chen
> Priority: Major
>
> In KAFKA-9751, when create internal topics through FindCoordinator or
> Metadata request, we route the topic creation request to the controller
> instead of handling by itself. We change logic in
> `KafkaApis#getTopicMetadata`, and make the internal topic won't get created
> when "auto.create.topics.enable=false`.
> h4.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
