Hi, Lianet, Thanks for the KIP.
JR1. It would be useful to provide a bit more motivation for the KIP. The batches allocated from the buffer pool are proportional to the number of active partitions. For publishing records without keys, the active partition is 1 by default, independent of the number of partitions in a topic. It's only when publishing records with keys that the active partition can be the total number of partitions in a topic. So, a possible scenario is that a client publishes records without keys to one topic while publishing records with keys to another. JR2. "Following records appended to the batch do not block or throw. They attempt non-blocking pool allocation and fall back to direct heap if the pool is exhausted. Ensures not blocking on pool memory while already holding some for a batch". Currently, the producer only allocates memory exceeding the configured buffer pool size in two cases. (1) Compressed data exceeding the estimated size (2) When a batch is too large for the broker's max.message.bytes and gets split, each sub-batch is allocated via ByteBuffer.allocate(initialSize) directly. With the KIP, are we introducing new cases in addition to the above two? Jun On Fri, May 1, 2026 at 6:03 AM Lianet Magrans <[email protected]> wrote: > Thanks for the feedback Jaisen! I like your proposed "static" for the > current behaviour, it aligns nicely. All updated. > > Best! > Lianet > > On Thu, Apr 30, 2026 at 4:27 PM Jaisen Mathai via dev < > [email protected]> > wrote: > > > Thanks Lianet. > > > > I like the proposal. > > > > I suggest a descriptive name such as static or fixed instead of legacy > for > > the default configuration value. I think these will age better while > still > > communicating that users should strongly consider using the non-default > > value of dynamic. > > > > Jaisen > > > > On Thu, Apr 30, 2026 at 8:02 AM Lianet Magrans <[email protected]> > wrote: > > > > > Hi all, > > > > > > I would like to start a discussion on KIP-1332 that proposes a dynamic > > > memory allocation strategy for the Kafka producer, to unlock > high-latency > > > scenarios increasingly common as Kafka moves toward object storage. > > > > > > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1332*3A*Dynamic*memory*allocation*for*the*Kafka*producer__;JSsrKysrKys!!Ayb5sqE7!t4yI-C5BwMxJ6dMJC7tuQhu94KuolbgKXyEnl4GChJGLYY2eS4NXk-GZYlnVPnuw3ESrGwKjyPDr5Bjp0Gk$ > > > > > > Thanks! > > > Lianet > > > > > >
