Hi Lianet, Thanks for this great KIP.
TYC1. I have one consideration regarding observability: Do we need a new metric for average-chunks-per-batch? With the introduction of the chunked-buffer strategy, memory usage per partition is no longer a fixed batch.size. While this significantly improves memory efficiency, it might be beneficial for operators to understand the actual "chunk utilization" or fragmentation under different workloads. Specifically, I think this metric would be valuable when combined with the proposed bufferpool-overflow metrics: it would help operators distinguish whether memory pressure is being driven by a large number of active partitions (many small batches) or by individual batches becoming unexpectedly large (many chunks per batch, perhaps due to large records or low compression ratios). What do you think? Best, TengYao Chi On 2026/05/11 23:03:53 Jun Rao via dev wrote: > Hi, Lianet, > > Thanks for the KIP. > > JR1. It would be useful to provide a bit more motivation for the KIP. The > batches allocated from the buffer pool are proportional to the number of > active partitions. For publishing records without keys, the active > partition is 1 by default, independent of the number of partitions in a > topic. It's only when publishing records with keys that the active > partition can be the total number of partitions in a topic. So, a possible > scenario is that a client publishes records without keys to one topic while > publishing records with keys to another. > > JR2. "Following records appended to the batch do not block or throw. They > attempt non-blocking pool allocation and fall back to direct heap if the > pool is exhausted. > Ensures not blocking on pool memory while already holding some for a batch". > > Currently, the producer only allocates memory exceeding the configured > buffer pool size in two cases. > (1) Compressed data exceeding the estimated size > (2) When a batch is too large for the broker's max.message.bytes and gets > split, each sub-batch is allocated via ByteBuffer.allocate(initialSize) > directly. > > With the KIP, are we introducing new cases in addition to the above two? > > Jun > > > > On Fri, May 1, 2026 at 6:03 AM Lianet Magrans <[email protected]> wrote: > > > Thanks for the feedback Jaisen! I like your proposed "static" for the > > current behaviour, it aligns nicely. All updated. > > > > Best! > > Lianet > > > > On Thu, Apr 30, 2026 at 4:27 PM Jaisen Mathai via dev < > > [email protected]> > > wrote: > > > > > Thanks Lianet. > > > > > > I like the proposal. > > > > > > I suggest a descriptive name such as static or fixed instead of legacy > > for > > > the default configuration value. I think these will age better while > > still > > > communicating that users should strongly consider using the non-default > > > value of dynamic. > > > > > > Jaisen > > > > > > On Thu, Apr 30, 2026 at 8:02 AM Lianet Magrans <[email protected]> > > wrote: > > > > > > > Hi all, > > > > > > > > I would like to start a discussion on KIP-1332 that proposes a dynamic > > > > memory allocation strategy for the Kafka producer, to unlock > > high-latency > > > > scenarios increasingly common as Kafka moves toward object storage. > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1332*3A*Dynamic*memory*allocation*for*the*Kafka*producer__;JSsrKysrKys!!Ayb5sqE7!t4yI-C5BwMxJ6dMJC7tuQhu94KuolbgKXyEnl4GChJGLYY2eS4NXk-GZYlnVPnuw3ESrGwKjyPDr5Bjp0Gk$ > > > > > > > > Thanks! > > > > Lianet > > > > > > > > > >
