Hi Michał, On Thu, Feb 27, 2025, at 3:44 AM, Michał Łowicki wrote: > Hi there! > > Is there any reason why Metadata requests > <https://kafka.apache.org/protocol.html#The_Messages_Metadata> do not > support fetching metadata for subsets of the partitions? If a certain > client is interested only in e.g. 1 but topic may have many so most of > fetched data isn't really used. >
It's certainly been a topic that's come up before. In certain situations the current approach is a bit heavy-handed. The current approach for fetching metadata has a number of benefits: it keeps the protocol from being too chatty, which reduces load on the brokers and makes maintaining a consistent via of the metadata on the client much easier. There's a fairly substantial overhead with fetching metadata and batching it in a single request eliminates a lot of edge cases. As always, further discussion and suggestions for improvements in this area are welcomed :) Thanks, Kirk
