On Thu, Feb 27, 2025 at 5:39 PM Kirk True <k...@kirktrue.pro> wrote:
> Hi Michał, > > On Thu, Feb 27, 2025, at 3:44 AM, Michał Łowicki wrote: > > Hi there! > > > > Is there any reason why Metadata requests > > <https://kafka.apache.org/protocol.html#The_Messages_Metadata> do not > > support fetching metadata for subsets of the partitions? If a certain > > client is interested only in e.g. 1 but topic may have many so most of > > fetched data isn't really used. > > > > It's certainly been a topic that's come up before. In certain situations > the current approach is a bit heavy-handed. The current approach for > fetching metadata has a number of benefits: it keeps the protocol from > being too chatty, which reduces load on the brokers and makes maintaining a > consistent via of the metadata on the client much easier. There's a fairly > substantial overhead with fetching metadata and batching it in a single > request eliminates a lot of edge cases. > Sure, I'm rather thinking about an opt-in option to the protocol where, if specified, metadata response would contain metadata for a specified set of partitions (otherwise as of today metadata for all of them). To cover the cases where consumers need to know metadata for only a small portion of partitions. Then it would be less for the broker to handle such requests and craft responses and protocol would be actually less chatty in those cases. > > As always, further discussion and suggestions for improvements in this > area are welcomed :) > > Thanks, > Kirk
