[
https://issues.apache.org/jira/browse/KAFKA-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634151#comment-14634151
]
Jason Gustafson commented on KAFKA-2275:
----------------------------------------
[~singhashish], thanks for the response. Comments below.
{quote}
Maybe I should move the flag, indicating cluster has metadata for all topics or
subset of topics, to Cluster. Makes sense?
{quote}
Yeah, that might work. I've also wondered if we could just keep a separate
Cluster object when querying for all metadata, but it feels like overkill. I
actually sort of think that we need to be able to send metadata requests
through NetworkClient without it hijacking the response. Then we wouldn't need
to worry about partitionsFor polluting the state of the consumer with metadata
we don't care about. Perhaps this could be done by having NetworkClient peek at
the in-flight requests to see if there is a pending metadata request instead of
just consuming the response directly.
{quote}
I do not think there is a way to directly subscribe to metadata changes as of
now. Correct me if my understanding is wrong. One would have to periodically
poll to get metadata updates. Now, the question becomes where should this
polling be done? With the current modification, the regex subscriber will have
to manage the polling logic. We can definitely push the polling logic down to
say Network client, but then the question will be is it required? Let me know
your thoughts.
{quote}
I think we can manage the polling with a background task (sort of like how
heartbeats and auto-commits are done). But if we're sort of concurrently
sending out requests for all topics and for only a subset of the topics, we'd
have to get a little lucky that the "right" metadata is available when the task
runs. Does that make sense?
> Add a ListTopics() API to the new consumer
> ------------------------------------------
>
> Key: KAFKA-2275
> URL: https://issues.apache.org/jira/browse/KAFKA-2275
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
> Reporter: Guozhang Wang
> Assignee: Ashish K Singh
> Priority: Critical
> Fix For: 0.8.3
>
> Attachments: KAFKA-2275.patch, KAFKA-2275_2015-07-17_21:39:27.patch,
> KAFKA-2275_2015-07-20_10:44:19.patch
>
>
> With regex subscription like
> {code}
> consumer.subscribe("topic*")
> {code}
> The partition assignment is automatically done at the Kafka side, while there
> are some use cases where consumers want regex subscriptions but not
> Kafka-side partition assignment, rather with their own specific partition
> assignment. With ListTopics() they can periodically check for topic list
> changes and specifically subscribe to the partitions of the new topics.
> For implementation, it involves sending a TopicMetadataRequest to a random
> broker and parse the response.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
