If you have thousands of topics and dozens of brokers, your metadata responses could easily get into the megabyte range. At that point it will start affecting your latency statistics.
Right now there is absolutely no way for management systems to work around this. They have to make a giant MetadataRequest or nothing at all. With the addition of ListTopicsRequest, management systems can get the full list, and then make a few MetadataRequests with a subset of the topics set. So you can use ListTopicsRequest to get all 3,500 topic names, and then make several MetadataRequests to get details for 100 topics at a time (or whatever batch size you want to use). Another application: It's pretty common to need to invoke list topics after deleting some topics. This lets you wait until the delete has finished. In this case, you should not use MetadataRequest with just a single topic name, since you may trigger topic auto-create. Right now your only choice is MetadataRequest(topics=*) And the simplest application... "./bin/kafka-topics.sh --list" returns just topic names, no other information by default. best, Colin On Thu, Apr 20, 2017, at 15:18, Jeff Widman wrote: > In theory this is nice. > > However, one possibly silly question... personally I've found that most > times when I wanted to list the topics, I normally also wanted to know > additional metadata such as the leader of the topic. > > So I just want to doublecheck that there are enough common use cases that > only want to know the list of topics to merit creating a new > request/response type? > > > > On Thu, Apr 20, 2017 at 2:17 PM, Colin McCabe <cmcc...@apache.org> wrote: > > > Hi all, > > > > While working on the AdminClient, we determined that we didn't have a > > way to efficiently list all the topics in the cluster. I wrote up a KIP > > to add a ListTopicsRequest to make this more efficient in larger > > clusters with more brokers and topics. Please take a look at the KIP > > here: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 142%3A+Add+ListTopicsRequest+to+efficiently+list+all+the+ > > topics+in+a+cluster > > > > cheers, > > Colin > >
