Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-12 Thread via GitHub


chia7712 commented on PR #21065:
URL: https://github.com/apache/kafka/pull/21065#issuecomment-3742126841

   @dnadolny Thanks for the fantastic contribution!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-12 Thread via GitHub


dnadolny commented on PR #21065:
URL: https://github.com/apache/kafka/pull/21065#issuecomment-3740826086

   @chia7712 yup looks good, thanks for taking it!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-11 Thread via GitHub


chia7712 commented on PR #21065:
URL: https://github.com/apache/kafka/pull/21065#issuecomment-3734432804

   4.2: https://github.com/apache/kafka/pull/21285
   4.1: https://github.com/apache/kafka/pull/21286
   4.0: https://github.com/apache/kafka/pull/21287
   3.9: https://github.com/apache/kafka/pull/21288
   
   @dnadolny I was bored, so I did this backport for fun (and to keep the 
release manager happy). Mind taking a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-10 Thread via GitHub


chia7712 merged PR #21065:
URL: https://github.com/apache/kafka/pull/21065


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-10 Thread via GitHub


chia7712 commented on PR #21065:
URL: https://github.com/apache/kafka/pull/21065#issuecomment-3732914372

   The flaky test is tracked in KAFKA-15529, so I have re-triggered the build


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-10 Thread via GitHub


chia7712 commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2678711503


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##
@@ -524,6 +534,7 @@ private void maybeAbortBatches(RuntimeException exception) {
 if (accumulator.hasIncomplete()) {
 log.error("Aborting producer batches due to fatal error", 
exception);
 accumulator.abortBatches(exception);
+inFlightBatches.clear();

Review Comment:
   nice!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-09 Thread via GitHub


junrao commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2677364176


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1157,14 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+if (batch.isInflight()) {
+// KAFKA-19012: if the batch has been sent it might still be 
in use by the network client so we cannot allow it to be reused yet.
+// We skip deallocating it now. When the request in network 
client completes with a response, either completeBatch() or 
+// failBatch() will be called with deallocateBatch=true. The 
buffer associated with the batch will be deallocated then.

Review Comment:
   It would be useful to make it clear that completeBatch() and failBatch() are 
in Sender.



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##
@@ -833,7 +851,18 @@ private void failBatch(
 log.debug("Encountered error when transaction manager was 
handling a failed batch", e);
 }
 }
-maybeRemoveAndDeallocateBatch(batch);
+if (deallocateBatch) {
+maybeRemoveAndDeallocateBatch(batch);
+} else {
+// Fix for KAFKA-19012
+// The pooled ByteBuffer associated with this batch might 
still be in use by the network client so we
+// cannot allow it to be reused yet. We skip deallocating it 
now. When the request in the network client 
+// completes with a response, either completeBatch() or 
failBatch() will be called with deallocateBatch=true.
+// The buffer associated with the batch will be deallocated 
then.
+maybeRemoveAndDeallocateBatchLater(batch);
+}
+} else {
+this.accumulator.deallocate(batch);

Review Comment:
   This is not an issue for now. But it will be useful to check deallocateBatch 
before deallocate().



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-09 Thread via GitHub


dnadolny commented on PR #21065:
URL: https://github.com/apache/kafka/pull/21065#issuecomment-3727688576

   @junrao Updated


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


dnadolny commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2674448044


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1044,6 +1044,9 @@ public void deallocate(ProducerBatch batch) {
 if (batch.isBufferDeallocated()) {
 log.warn("Skipping deallocating a batch that has already been 
deallocated. Batch is {}, created time is {}", batch, batch.createdMs);
 } else {
+if (batch.isInflight()) {
+throw new IllegalStateException("Attempting to deallocate 
a batch that is not inflight. Batch is " + batch);

Review Comment:
   If we just log a warning I worry it won't get caught in tests if there's a 
regression. One alternative is using an assert like 
https://github.com/apache/kafka/pull/21065#discussion_r2674253553, though how 
do you feel about deallocating a fresh buffer followed by throwing an 
exception? That way there's no memory leak, we fail tests if there's a 
regression, and get a stack trace in the logs?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


junrao commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2674334760


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1155,14 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+if (batch.isInflight()) {
+completeBatch(batch);
+} else {
+// KAFKA-19012: if the batch has been sent it might still be 
in use by the network client so we cannot allow it to be reused yet.
+// We skip deallocating it now. When the request in network 
client completes with a response, either completeBatch() or 
+// failBatch() will be called with deallocateBatch=true. The 
buffer associated with the batch will be deallocated then.

Review Comment:
   Could we move the comment to the if clause?



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1044,6 +1044,9 @@ public void deallocate(ProducerBatch batch) {
 if (batch.isBufferDeallocated()) {
 log.warn("Skipping deallocating a batch that has already been 
deallocated. Batch is {}, created time is {}", batch, batch.createdMs);
 } else {
+if (batch.isInflight()) {
+throw new IllegalStateException("Attempting to deallocate 
a batch that is not inflight. Batch is " + batch);

Review Comment:
   Since the Sender thread just logs an exception and continues, we could be 
silently leaking memory if this happens. An alternative is to log a warn and 
create a fresh buffer of batch.initialCapacity() size to deallocate. It would 
be useful to include the stacktrace in the log message. 
   
   > that is not inflight
   
   that is inflight



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


dnadolny commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2674253553


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1027,14 +1027,34 @@ BuiltInPartitioner createBuiltInPartitioner(LogContext 
logContext, String topic,
 }
 
 /**
- * Deallocate the record batch
+ * Complete and deallocate the record batch
+ */
+public void completeAndDeallocateBatch(ProducerBatch batch) {
+completeBatch(batch);
+deallocate(batch);
+}
+
+/**
+ * Only perform deallocation (and not removal from the incomplete set)
  */
 public void deallocate(ProducerBatch batch) {
-incomplete.remove(batch);
 // Only deallocate the batch if it is not a split batch because split 
batch are allocated outside the
 // buffer pool.
-if (!batch.isSplitBatch())
-free.deallocate(batch.buffer(), batch.initialCapacity());
+if (!batch.isSplitBatch()) {
+if (batch.isBufferDeallocated()) {
+log.warn("Skipping deallocating a batch that has already been 
deallocated. Batch is {}, created time is {}", batch, batch.createdMs);
+} else {
+batch.markBufferDeallocated();

Review Comment:
   I've added it as `IllegalStateException`, though this means that we'll leak 
memory if it happens - if we throw an exception here, we may never deallocate 
the batch. The total amount of memory leaked vs heap isn't bad, but if the 
BufferPool runs low against its configured size it slows down / blocks 
indefinitely.
   An alternative would be to do it as an `assert`, which runs during tests but 
aren't on by default as far as I know when running for real. This won't leak 
memory, but allows the potential for corruption.
   I think the latter is worse, but wanted to bring it up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


dnadolny commented on PR #21065:
URL: https://github.com/apache/kafka/pull/21065#issuecomment-3726376839

   @junrao Updated the code


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


dnadolny commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2674253553


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1027,14 +1027,34 @@ BuiltInPartitioner createBuiltInPartitioner(LogContext 
logContext, String topic,
 }
 
 /**
- * Deallocate the record batch
+ * Complete and deallocate the record batch
+ */
+public void completeAndDeallocateBatch(ProducerBatch batch) {
+completeBatch(batch);
+deallocate(batch);
+}
+
+/**
+ * Only perform deallocation (and not removal from the incomplete set)
  */
 public void deallocate(ProducerBatch batch) {
-incomplete.remove(batch);
 // Only deallocate the batch if it is not a split batch because split 
batch are allocated outside the
 // buffer pool.
-if (!batch.isSplitBatch())
-free.deallocate(batch.buffer(), batch.initialCapacity());
+if (!batch.isSplitBatch()) {
+if (batch.isBufferDeallocated()) {
+log.warn("Skipping deallocating a batch that has already been 
deallocated. Batch is {}, created time is {}", batch, batch.createdMs);
+} else {
+batch.markBufferDeallocated();

Review Comment:
   I've added it as `IllegalStateException`, though this means that we'll leak 
memory if it happens - if we throw an exception here, we may never deallocate 
the batch. The total amount of memory leaked vs heap isn't bad, but if if the 
BufferPool runs low against its configured size it slows down / blocks 
indefinitely.
   An alternative would be to do it as an `assert`, which runs during tests but 
aren't on by default as far as I know when running for real. This won't leak 
memory, but allows the potential for corruption.
   I think the latter is worse, but wanted to bring it up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


dnadolny commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2674243064


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1152,14 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+if (batch.isSent()) {

Review Comment:
   You mean just this? 
https://github.com/apache/kafka/pull/21065/commits/da6d092ec394e1eff0424f9effd3ba47639fface



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


dnadolny commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2674241910


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerBatch.java:
##
@@ -72,6 +72,8 @@ private enum FinalState { ABORTED, FAILED, SUCCEEDED }
 private final AtomicInteger attempts = new AtomicInteger(0);
 private final boolean isSplitBatch;
 private final AtomicReference finalState = new 
AtomicReference<>(null);
+private boolean bufferDeallocated = false;
+private boolean sent = false;

Review Comment:
   Yup I flipped back and forth between making this inflight vs sent, initially 
thought it'd be easier to just mark sent rather than tracking whether it was 
inflight. Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


junrao commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2674049829


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1027,14 +1027,34 @@ BuiltInPartitioner createBuiltInPartitioner(LogContext 
logContext, String topic,
 }
 
 /**
- * Deallocate the record batch
+ * Complete and deallocate the record batch
+ */
+public void completeAndDeallocateBatch(ProducerBatch batch) {
+completeBatch(batch);
+deallocate(batch);
+}
+
+/**
+ * Only perform deallocation (and not removal from the incomplete set)
  */
 public void deallocate(ProducerBatch batch) {
-incomplete.remove(batch);
 // Only deallocate the batch if it is not a split batch because split 
batch are allocated outside the
 // buffer pool.
-if (!batch.isSplitBatch())
-free.deallocate(batch.buffer(), batch.initialCapacity());
+if (!batch.isSplitBatch()) {
+if (batch.isBufferDeallocated()) {
+log.warn("Skipping deallocating a batch that has already been 
deallocated. Batch is {}, created time is {}", batch, batch.createdMs);
+} else {
+batch.markBufferDeallocated();

Review Comment:
   Could we also verify that batch.sent is false and if not, throw an 
IllegalStateException?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


junrao commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673999688


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   @chia7712 : Yes, that's the potential issue.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


jolshan commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673961833


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   @chia7712 
   One thing to keep in mind is that the newer version of the transactional 
protocol doesn't add the partition separately. The addition is done during the 
send. 
   
   There is also a case to consider where the partitions have already been 
added to the transaction and we skip straight to sending the produce request. 
   
   In either scenario though, I think this analysis is roughly the same minus 
the add partitions requests (if we enter fatal state for some other reason)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


jolshan commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673961833


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   @chia7712 
   One thing to keep in mind is that the newer version of the transactional 
protocol doesn't add the partition first. The addition is done during the send. 
   
   There is also a case to consider where the partitions have already been 
added to the transaction and we skip straight to sending the produce request. 
   
   In either scenario though, I think this analysis is roughly the same minus 
the add partitions requests (if we enter fatal state for some other reason)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


chia7712 commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673822421


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   > However, the record will first be added to the batch buffer in 
RecordAccumulator and then aborted by Sender. Adding to the RecordAccumulator 
can cause the buffer reuse issue.
   
   1. Trigger Request 1: Send a record to p0. This initiates an 
AddPartitionsToTxnRequest (for p0) and stages a ProducerRequest.
   2. Trigger Request 2: Send a record to p1. This initiates an 
AddPartitionsToTxnRequest (for p1).
   3. In-flight State: AddPartitionsToTxnRequest(p0) succeeds. The 
corresponding ProducerRequest(p0) is sent and becomes in-flight.
   4. Fatal Error Trigger: AddPartitionsToTxnRequest(p1) fails with an 
authorization error. The TxnManager transitions to a fatal state and triggers 
abortBatches.
   5. Unsafe Deallocation: batch(p0) is **prematurely deallocated** by 
abortBatches while still in-flight.
   6. Memory Pollution: The application sends a record to px. This new batch 
acquires the dangling buffer from batch(p0) and overwrites the memory.
   7. Corrupted Send: The NetworkClient, still holding the reference to the 
original buffer, sends the now-corrupted data to p0.
   
   Please let me know if I missed anything in this analysis



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


junrao commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673763604


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   > Given that Producer.send throws an AuthorizationException when the 
TxnManager is in a fatal state, and the standard recommendation is to close the 
producer immediately for such unrecoverable errors, the practical risk should 
be minimal
   
   It's true that Producer.send throws an AuthorizationException when the 
TxnManager is in a fatal state. However, the record will first be added to the 
batch buffer in RecordAccumulator and then aborted by Sender. Adding to the 
RecordAccumulator can cause the buffer reuse issue.
   
   > I think the only concern I had is whether a message could be sent to a 
partition and read by a consumer
   
   I think that the record with the wrong data could be sent to a partition. 
Since the TxnManager is in a fatal state, the wrong record will probably be 
aborted on the server side eventually? If the consumer is in transactional 
mode, it won't see the wrong record. However, if the consumer is in 
non-transactional mode, it will see it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


junrao commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673633951


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1152,14 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+if (batch.isSent()) {

Review Comment:
   This is not a real issue, but it would safer for the caller 
`Sender.maybeAbortBatches()` to clear `inFlightBatches` after aborting all 
batches.



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerBatch.java:
##
@@ -581,6 +583,22 @@ public boolean sequenceHasBeenReset() {
 return reopened;
 }
 
+public boolean isBufferDeallocated() {
+return bufferDeallocated;
+}
+
+public void markBufferDeallocated() {
+bufferDeallocated = true;
+}
+
+public boolean isSent() {
+return sent;
+}
+
+public void markSent() {

Review Comment:
   It would be useful to add a method to mark the batch as unsent when the 
request completes with a response. This is important since a batch could be 
re-enqueued for retry. It would be useful to verify in the re-enqueue test that 
the batch is marked as unsent after re-enqueuing. 



##
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java:
##
@@ -2195,6 +2199,8 @@ public void testCancelInFlightRequestAfterFatalError() 
throws Exception {
 Future future2 = appendToAccumulator(tp1);
 sender.runOnce();
 
+assertFalse(pool.allMatch());

Review Comment:
   Could we do the same check just before the 2nd produce response completes? 
When I added that, it exposed the bug in RecordAccumulator.



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerBatch.java:
##
@@ -72,6 +72,8 @@ private enum FinalState { ABORTED, FAILED, SUCCEEDED }
 private final AtomicInteger attempts = new AtomicInteger(0);
 private final boolean isSplitBatch;
 private final AtomicReference finalState = new 
AtomicReference<>(null);
+private boolean bufferDeallocated = false;
+private boolean sent = false;

Review Comment:
   Could we add a comment that this means that the batch is sent to the 
NetworkClient? Also, probably inflight is a better name since it's more 
consistent with Sender.inflights.



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   > The problem is that the newly accumulated batches (that will get aborted 
on runOnce()) can modify the infight batch? And then the problem is that we may 
send the inflight batch to the wrong place?
   
   When the TxnManager gets into the fatal state, there could be an inflight 
batch in the NetworkClient. maybeAbortBatches() will return the buffer 
associated with the inflight batch to buffer pool. RecordAccumulator.append() 
is allowed when TxnManager is in the fatal state and it could put new records 
into the just returned batch buffer. Sender continues to call client.poll() and 
triggers the sending of the inflight request, which will pick up the wrong data 
in the batch buffer. The changes in 
SenderTest.testCancelInFlightRequestAfterFatalError() simulates the issue.



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1152,14 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+if (batch.isSent()) {
+completeAndDeallocateBatch(batch);

Review Comment:
   This is a bug. If the batch is sent, we shouldn't deallocate the batch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


jolshan commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673692487


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   I think the only concern I had is whether a message could be sent to a 
partition and read by a consumer



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


chia7712 commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673675990


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   > the remaining question is whether the buffer reuse issue could happen when 
the TxnManager is in the fatal state.
   
   Given that `Producer.send` throws an `AuthorizationException` when the 
`TxnManager` is in a fatal state, and the standard recommendation is to close 
the producer immediately for such unrecoverable errors, the practical risk 
should be minimal  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


jolshan commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673520340


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   Can you clarify this part -- apologies, I may just not understand the bug 
fully
   
   > My understanding is that if TxnManager is in the fatal state, producer 
could still accumulate record batches in the accumulator. The Sender will just 
abort those batches in runOnce(). This could still expose the buffer reuse 
problem.
   
   The problem is that the newly accumulated batches (that will get aborted on 
runOnce()) can modify the infight batch? And then the problem is that we may 
send the inflight batch to the wrong place? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


junrao commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673470117


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   For the maybeAbortBatches() call in the following method, we only call that 
when the initPid call failed. At that point, no batches could be sent. So, it 
doesn't seem this exposes the buffer reuse issue.
   
   ```
   private boolean shouldHandleAuthorizationError(RuntimeException 
exception) {
   if (exception instanceof TransactionalIdAuthorizationException ||
   exception instanceof ClusterAuthorizationException) {
   transactionManager.failPendingRequests(new 
AuthenticationException(exception));
   maybeAbortBatches(exception);
   transactionManager.transitionToUninitialized(exception);
   return true;
   }
   return false;
   }
   ```
   
   So, the remaining question is whether the buffer reuse issue could happen 
when the TxnManager is in the fatal state.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


junrao commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673349295


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   > Are these solutions mostly to indicate correctly what we should do with 
the buffers and not reuse them?
   
   Yes, we just want to make sure if a batch is inflight in the NetworkClient, 
the deallocation of the buffer associated with the batch needs to wait until 
the inflight request completes if the batch is aborted early.
   
   > did you mean fatal errors or abortable here? If it is fatal, we also can't 
transactionally produce anymore.
   
   My understanding is that if TxnManager is in the fatal state, producer could 
still accumulate record batches in the accumulator. The Sender will just abort 
those batches in runOnce(). This could still expose the buffer reuse problem.
   
   ```
   // do not continue sending if the transaction manager is in 
a failed state
   if (transactionManager.hasFatalError()) {
   if (lastError != null)
   maybeAbortBatches(lastError);
   ```
   
   Also, there is a separate path calling maybeAbortBatches() when the 
TxnManager is in the abortable state. The producer is expected to continue 
afterwards.
   
   ```
   private boolean shouldHandleAuthorizationError(RuntimeException 
exception) {
   if (exception instanceof TransactionalIdAuthorizationException ||
   exception instanceof ClusterAuthorizationException) {
   transactionManager.failPendingRequests(new 
AuthenticationException(exception));
   maybeAbortBatches(exception);
   transactionManager.transitionToUninitialized(exception);
   return true;
   }
   return false;
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


junrao commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673349295


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   > Are these solutions mostly to indicate correctly what we should do with 
the buffers and not reuse them?
   
   Yes, we just want to make sure if a batch is inflight in the NetworkClient, 
the deallocation of the buffer associated with the batch needs to wait until 
the inflight request completes if the batch is aborted early.
   
   > did you mean fatal errors or abortable here? If it is fatal, we also can't 
transactionally produce anymore.
   
   My understanding is that if TxnManager is in the fatal state, producer could 
still accumulate record batches in the accumulator. The Sender will just abort 
those batches in runOnce(). This could still expose the buffer reuse problem.
   
   Also, there is a separate path calling maybeAbortBatches() when the 
TxnManager is in the abortable state. The producer is expected to continue 
afterwards.
   
   ```
   private boolean shouldHandleAuthorizationError(RuntimeException 
exception) {
   if (exception instanceof TransactionalIdAuthorizationException ||
   exception instanceof ClusterAuthorizationException) {
   transactionManager.failPendingRequests(new 
AuthenticationException(exception));
   maybeAbortBatches(exception);
   transactionManager.transitionToUninitialized(exception);
   return true;
   }
   return false;
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


jolshan commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673296982


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   
https://github.com/apache/kafka/blob/2f90af70ee61dab3dae43a82d29f9f8505056bc9/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L313



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


jolshan commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673288472


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   One clarification though -- did you mean fatal errors or abortable here? If 
it is fatal, we also can't transactionally produce anymore.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


jolshan commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673283612


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   Sorry a bit late to the party -- but yes, we could have messages already 
sent batches to my understanding. 
   
   For the fix: 
   > I am thinking that we can fix this in the following way. In 
abortBatches(), if a batch is in-flight, we call completeBatch() and otherwise, 
call completeBatchAndDeallocate(). We can add a boolean in ProducerBatch to 
indicate whether it's in-flight or not. The in-flight batches are already 
tracked in Sender. But adding a flag in ProducerBatch is probably more 
convenient.
   
   Are these solutions mostly to indicate correctly what we should do with the 
buffers and not reuse them? I'm still trying to fully understand what the 
result here would look like for a transactional producer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


jolshan commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673288472


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   One clarification though -- did you mean fatal errors or abortable here? If 
it is fatal, we also can't transactionally produce anymore to my understanding.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


jolshan commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673283612


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   Sorry a bit late to the party -- but yes, we could have messages already 
sent batches to my understanding. This is why it is important to abort.
   
   For the fix: 
   > I am thinking that we can fix this in the following way. In 
abortBatches(), if a batch is in-flight, we call completeBatch() and otherwise, 
call completeBatchAndDeallocate(). We can add a boolean in ProducerBatch to 
indicate whether it's in-flight or not. The in-flight batches are already 
tracked in Sender. But adding a flag in ProducerBatch is probably more 
convenient.
   
   Are these solutions mostly to indicate correctly what we should do with the 
buffers and not reuse them? I'm still trying to fully understand what the 
result here would look like for a transactional producer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-08 Thread via GitHub


jolshan commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2673283612


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   Sorry a bit late to the party -- but yes, we could have messages already 
sent batches to my understanding. 
   
   For the fix: 
   > I am thinking that we can fix this in the following way. In 
abortBatches(), if a batch is in-flight, we call completeBatch() and otherwise, 
call completeBatchAndDeallocate(). We can add a boolean in ProducerBatch to 
indicate whether it's in-flight or not. The in-flight batches are already 
tracked in Sender. But adding a flag in ProducerBatch is probably more 
convenient.
   
   Are these solutions mostly to indicate correctly what we should do with the 
buffers and not reuse them? I'm still trying to fully understand what the 
result here would look like for a transactional producer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-07 Thread via GitHub


junrao commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2670500662


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##
@@ -833,7 +849,17 @@ private void failBatch(
 log.debug("Encountered error when transaction manager was 
handling a failed batch", e);
 }
 }
-maybeRemoveAndDeallocateBatch(batch);
+if (deallocateBatch) {
+maybeRemoveAndDeallocateBatch(batch);
+} else {
+// Fix for KAFKA-19012
+// The pooled ByteBuffer associated with this batch might 
still be in use by the network client so we
+// cannot allow it to be reused yet. We skip deallocating it 
now, instead doing that in completeBatch
+// or in another run of failBatch where we hit the else clause 
and call deallocateAlreadyRemovedIncomplete
+maybeRemoveAndDeallocateBatchLater(batch);

Review Comment:
   > We skip deallocating it now, instead doing that in completeBatch
   >  or in another run of failBatch where we hit the else clause and call 
deallocateAlreadyRemovedIncomplete
   
   How about "We skip deallocating it now. When the request in network client 
completes with a response, either completeBatch() or failBatch() will be called 
with deallocateBatch=true. The buffer associated with the batch will be 
deallocated then."?



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerBatch.java:
##
@@ -72,6 +72,7 @@ private enum FinalState { ABORTED, FAILED, SUCCEEDED }
 private final AtomicInteger attempts = new AtomicInteger(0);
 private final boolean isSplitBatch;
 private final AtomicReference finalState = new 
AtomicReference<>(null);
+private boolean bufferDeallocated;

Review Comment:
   Could we explicitly initialize it to false?



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerBatch.java:
##
@@ -581,6 +582,14 @@ public boolean sequenceHasBeenReset() {
 return reopened;
 }
 
+public boolean isBufferDeallocated() {
+return bufferDeallocated;
+}
+
+public void setBufferDeallocated(boolean bufferDeallocated) {

Review Comment:
   Could we name this method markBufferDeallocated and remove the input param?



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1029,12 +1029,36 @@ BuiltInPartitioner createBuiltInPartitioner(LogContext 
logContext, String topic,
 /**
  * Deallocate the record batch
  */
-public void deallocate(ProducerBatch batch) {
+public void completeBatchAndDeallocate(ProducerBatch batch) {

Review Comment:
   completeBatchAndDeallocate => completeAndDeallocateBatch to match what's in 
RecordAccumulator ?



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   Thanks for doing the test. abortBatches() is called in the following 2 cases.
   1. When the producer is closed (related to the failure in 
SenderTest.testForceCloseWithProducerIdReset).  This is not causing a real 
issue since the networkClient won't be sending new requests after closing.
   2. When the producer's txnManager is in fatal state (related to the failure 
in SenderTest.testCancelInFlightRequestAfterFatalError) or when handling 
authorization error. This could be a real issue.
   
   I am thinking that we can fix this in the following way. In abortBatches(), 
if a batch is in-flight, we call completeBatch() and otherwise, call 
completeBatchAndDeallocate(). We can add a boolean in ProducerBatch to indicate 
whether it's in-flight or not. The in-flight batches are already tracked in 
Sender. But adding a flag in ProducerBatch is probably more convenient.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-07 Thread via GitHub


dnadolny commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2658098950


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java:
##
@@ -274,6 +274,17 @@ public void deallocate(ByteBuffer buffer, int size) {
 }
 }
 
+/**
+ * buffer is passed in only for use in tests
+ */
+public void deallocateWithoutReuse(ByteBuffer buffer, int size) {
+// We pass in a fresh buffer to be deallocated (added back to the pool 
if it is a poolable size)
+// This is slightly inefficient if it's not a poolable size, but those 
tend to be small and
+// this code is not hit often anyway
+// This is useful if we're not 100% sure the buffer is not in use, 
which happens in some error cases
+deallocate(ByteBuffer.allocate(size));

Review Comment:
   Not 100% sure this is what you intended but I gave it a shot. The code is 
getting a bit uglier, but what I've done is:
- in Sender we treat expired in-flight batches differently than before, all 
other batches the state and deallocation is how it was before
- for expired in-flight batches, we do all the regular logic when it used 
to happen except for calling deallocate in our BufferPool, which we defer until 
later
- we keep track of those expired batches that we haven't deallocated yet 
and when we hit the request timeout, we do the BufferPool free step. I didn't 
see an existing place in the code to hook in to in order to do this so I've 
done it in the new method `deallocateRequestTimeoutBatches`
   
   Note that I've kept the logic of having the BufferPool create a fresh 
ByteBuffer here, it's possible it's no longer necessary with the logic change 
above but my reasoning for keeping it is:
- it guarantees that we've fixed the corruption issue
- if the request timeout stuff all happens as expected then all we've done 
is be a tiny bit inefficient by creating a new ByteBuffer in a pretty rare 
case, but if there's any slight variation in timing then what would have been 
corruption is instead just using slightly more memory than we should have for a 
very short period of time



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-07 Thread via GitHub


dnadolny commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2668954861


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   At the least it can happen in tests. I [added a 
flag](https://github.com/dnadolny/kafka/commit/42693053658faf72b1dbda0b825aa3cd5385d719)
 to ProducerBatch and set it in Sender.sendProduceRequest, and then at this 
location in RecordAccumulator I throw an exception if it's set and it gets hit 
in `SenderTest.testCancelInFlightRequestAfterFatalError` as well as 
`SenderTest.testForceCloseWithProducerIdReset` 
   
   If this is a real issue it might be worth adding some logic like that to run 
all the time, to validate that deallocate is only called on batches that are 
either not sent or are already complete.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-07 Thread via GitHub


dnadolny commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2668954861


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   At the least it can happen in tests. I added a flag to ProducerBatch and set 
it in Sender.sendProduceRequest, and then at this location in RecordAccumulator 
I throw an exception if it's set and it gets hit in 
SenderTest.testCancelInFlightRequestAfterFatalError as well as 
SenderTest.testForceCloseWithProducerIdReset
   
   If this is a real issue it might be worth adding some logic like that to run 
all the time, to validate that deallocate is only called on batches that are 
either not sent or are already complete.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-07 Thread via GitHub


dnadolny commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2668887846


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##
@@ -749,12 +759,16 @@ private void completeBatch(ProducerBatch batch, 
ProduceResponse.PartitionRespons
 
 if (batch.complete(response.baseOffset, response.logAppendTime)) {
 maybeRemoveAndDeallocateBatch(batch);
+} else {
+// Always safe to call dellocate because the batch keeps track of 
whether or not it was deallocated yet

Review Comment:
   Oh man I wish I could blame it on having a new keyboard or something like 
that. Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-06 Thread via GitHub


junrao commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2666548111


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1029,12 +1029,36 @@ BuiltInPartitioner createBuiltInPartitioner(LogContext 
logContext, String topic,
 /**
  * Deallocate the record batch
  */
-public void deallocate(ProducerBatch batch) {
+public void completeBatchAndDeallocate(ProducerBatch batch) {
 incomplete.remove(batch);
+deallocateIfNecessary(batch);
+}
+
+private void deallocateIfNecessary(ProducerBatch batch) {
 // Only deallocate the batch if it is not a split batch because split 
batch are allocated outside the
 // buffer pool.
-if (!batch.isSplitBatch())
-free.deallocate(batch.buffer(), batch.initialCapacity());
+if (!batch.isSplitBatch()) {
+if (batch.isBufferDeallocated()) {
+log.warn("Skipping deallocating a batch that has already been 
deallocated. This is expected to happen rarely: for expired in-flight batches. 
Batch is {}, created time is {}", batch, batch.createdMs);

Review Comment:
   > This is expected to happen rarely: for expired in-flight batches
   
   Calling deallocate() on the same batch more than once is not expected. For 
inflight batches, deallocate() should only be called when the ProduceRequest 
completes.



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1029,12 +1029,36 @@ BuiltInPartitioner createBuiltInPartitioner(LogContext 
logContext, String topic,
 /**
  * Deallocate the record batch
  */
-public void deallocate(ProducerBatch batch) {
+public void completeBatchAndDeallocate(ProducerBatch batch) {

Review Comment:
   Could we adjust the javadoc above to "Complete and deallocate the record 
batch"?



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##
@@ -749,12 +759,16 @@ private void completeBatch(ProducerBatch batch, 
ProduceResponse.PartitionRespons
 
 if (batch.complete(response.baseOffset, response.logAppendTime)) {
 maybeRemoveAndDeallocateBatch(batch);
+} else {
+// Always safe to call dellocate because the batch keeps track of 
whether or not it was deallocated yet

Review Comment:
   typo dellocate 



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1132,7 +1156,7 @@ void abortBatches(final RuntimeException reason) {
 dq.remove(batch);
 }
 batch.abort(reason);
-deallocate(batch);
+completeBatchAndDeallocate(batch);

Review Comment:
   Hmm, this is an existing issue and I am not sure if this is a real issue or 
not. This method is called when the TxnManager is in certain error state. The 
problem is that it completes and deallocates both unsent and sent batches. If 
we deallocate sent batches and the producer is able to send new records 
afterward, the deallocated buffer could be reused when the inflight sent may 
not have completed, leading to the same issue in the jira.
   
   @jolshan and @artemlivshits : When this method is called, is it possible for 
`incomplete` to contain sent batches? After this method is called, could the 
producer continue send new records?



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1029,12 +1029,36 @@ BuiltInPartitioner createBuiltInPartitioner(LogContext 
logContext, String topic,
 /**
  * Deallocate the record batch
  */
-public void deallocate(ProducerBatch batch) {
+public void completeBatchAndDeallocate(ProducerBatch batch) {
 incomplete.remove(batch);
+deallocateIfNecessary(batch);
+}
+
+private void deallocateIfNecessary(ProducerBatch batch) {

Review Comment:
   Could we just fold this into `deallocate()` and remove 
`deallocateIfNecessary()`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-06 Thread via GitHub


dnadolny commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2666498670


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##
@@ -749,12 +759,16 @@ private void completeBatch(ProducerBatch batch, 
ProduceResponse.PartitionRespons
 
 if (batch.complete(response.baseOffset, response.logAppendTime)) {
 maybeRemoveAndDeallocateBatch(batch);
+} else {
+// Always safe to call dellocate because the batch keeps track of 
whether or not it was deallocated yet
+this.accumulator.deallocateAlreadyRemovedIncomplete(batch);
 }
 }
 
 private void failBatch(ProducerBatch batch,
ProduceResponse.PartitionResponse response,
-   boolean adjustSequenceNumbers) {
+   boolean adjustSequenceNumbers,
+   boolean deallocateBatch) {

Review Comment:
   Even though it's redundant in that one, I'm hesitant to remove it because 
it's already pretty complex with multiple overloaded methods with multiple 
booleans.
   I think it's a bit less error prone to always have the flag in there even 
though that overloaded version is always called with true, but if you'd rather 
remove it I'll make the change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-06 Thread via GitHub


dnadolny commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2666498670


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##
@@ -749,12 +759,16 @@ private void completeBatch(ProducerBatch batch, 
ProduceResponse.PartitionRespons
 
 if (batch.complete(response.baseOffset, response.logAppendTime)) {
 maybeRemoveAndDeallocateBatch(batch);
+} else {
+// Always safe to call dellocate because the batch keeps track of 
whether or not it was deallocated yet
+this.accumulator.deallocateAlreadyRemovedIncomplete(batch);
 }
 }
 
 private void failBatch(ProducerBatch batch,
ProduceResponse.PartitionResponse response,
-   boolean adjustSequenceNumbers) {
+   boolean adjustSequenceNumbers,
+   boolean deallocateBatch) {

Review Comment:
   Even though it's redundant in that one, I'm hesitant to remove it because 
there are multiple overloaded methods with multiple booleans.
   I think it's a bit less error prone to always have the flag in there even 
though that overloaded version is always called with true, but if you'd rather 
remove it I'll make the change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-06 Thread via GitHub


dnadolny commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2666491549


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1031,12 +1031,30 @@ BuiltInPartitioner createBuiltInPartitioner(LogContext 
logContext, String topic,
  */
 public void deallocate(ProducerBatch batch) {
 incomplete.remove(batch);
+deallocateIfNecessary(batch);
+}
+
+private void deallocateIfNecessary(ProducerBatch batch) {
 // Only deallocate the batch if it is not a split batch because split 
batch are allocated outside the
 // buffer pool.
-if (!batch.isSplitBatch())
+if (!batch.isSplitBatch() && !batch.isBufferDeallocated() && 
batch.setBufferDeallocated(true))

Review Comment:
   The condition is a bit simpler now, though it's also expanded with logging



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-06 Thread via GitHub


dnadolny commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2666490172


##
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java:
##
@@ -3591,6 +3596,44 @@ public void 
testWhenProduceResponseReturnsWithALeaderShipChangeErrorAndNewLeader
 }
 }
 
+@Test
+public void testNoBufferReuseWhenBatchHasTimeoutErrors() throws Exception {
+long totalSize = 1024 * 1024;
+try (Metrics m = new Metrics()) {
+BufferPool pool = new BufferPool(totalSize, batchSize, m, time, 
"producer-internal-metrics");
+
+// Allocate and store a poolable buffer, then return it to the 
pool so the Sender can pick it up
+ByteBuffer buffer = pool.allocate(batchSize, 0);
+pool.deallocate(buffer);
+
+setupWithTransactionState(null, false, pool);
+StringBuilder sb = new StringBuilder();
+// Make a value the right size so that RecordAccumulator calls 
BufferPool.allocate with the batchSize so it's pooled
+for (int i = 0; i < 16290; i++) {
+sb.append("a");
+}

Review Comment:
   Ah for some reason I thought I had to make a message >= pooled size to get 
it to use the pool, but I retested and see that you're right and I don't need 
that. I've just changed it to `"value"`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-06 Thread via GitHub


chia7712 commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2666357603


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##
@@ -749,12 +759,16 @@ private void completeBatch(ProducerBatch batch, 
ProduceResponse.PartitionRespons
 
 if (batch.complete(response.baseOffset, response.logAppendTime)) {
 maybeRemoveAndDeallocateBatch(batch);
+} else {
+// Always safe to call dellocate because the batch keeps track of 
whether or not it was deallocated yet
+this.accumulator.deallocateAlreadyRemovedIncomplete(batch);
 }
 }
 
 private void failBatch(ProducerBatch batch,
ProduceResponse.PartitionResponse response,
-   boolean adjustSequenceNumbers) {
+   boolean adjustSequenceNumbers,
+   boolean deallocateBatch) {

Review Comment:
   There are two overloaded `failBatch` methods. the one with the redundant 
argument is the second definition 
   
   ```java
   private void failBatch(
   ProducerBatch batch,
   RuntimeException topLevelException,
   boolean adjustSequenceNumbers,
   boolean deallocateBatch
   )
   ```
   
   ```java
   private void failBatch(ProducerBatch batch,
  ProduceResponse.PartitionResponse response,
  boolean adjustSequenceNumbers,
  boolean deallocateBatch)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-06 Thread via GitHub


junrao commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2666314886


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##
@@ -749,12 +759,16 @@ private void completeBatch(ProducerBatch batch, 
ProduceResponse.PartitionRespons
 
 if (batch.complete(response.baseOffset, response.logAppendTime)) {
 maybeRemoveAndDeallocateBatch(batch);
+} else {
+// Always safe to call dellocate because the batch keeps track of 
whether or not it was deallocated yet
+this.accumulator.deallocateAlreadyRemovedIncomplete(batch);
 }
 }
 
 private void failBatch(ProducerBatch batch,
ProduceResponse.PartitionResponse response,
-   boolean adjustSequenceNumbers) {
+   boolean adjustSequenceNumbers,
+   boolean deallocateBatch) {

Review Comment:
   The following call passes in deallocateBatch as false.
   
   `failExpiredBatches(expiredInflightBatches, now, false);`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-06 Thread via GitHub


chia7712 commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2665898730


##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1031,12 +1031,30 @@ BuiltInPartitioner createBuiltInPartitioner(LogContext 
logContext, String topic,
  */
 public void deallocate(ProducerBatch batch) {
 incomplete.remove(batch);
+deallocateIfNecessary(batch);
+}
+
+private void deallocateIfNecessary(ProducerBatch batch) {
 // Only deallocate the batch if it is not a split batch because split 
batch are allocated outside the
 // buffer pool.
-if (!batch.isSplitBatch())
+if (!batch.isSplitBatch() && !batch.isBufferDeallocated() && 
batch.setBufferDeallocated(true))

Review Comment:
   WDYT about simplifying this logic like this?
   
   ```java
   public boolean tryMarkDeallocated() {
   return this.bufferDeallocated.compareAndSet(false, true);
   }
   
   private void deallocateIfNecessary(ProducerBatch batch) {
   // Only deallocate the batch if it is not a split batch because split 
batch are allocated outside the
   // buffer pool.
   if (!batch.isSplitBatch() && batch.tryMarkDeallocated())
   free.deallocate(batch.buffer(), batch.initialCapacity());
   }
   ```



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java:
##
@@ -749,12 +759,16 @@ private void completeBatch(ProducerBatch batch, 
ProduceResponse.PartitionRespons
 
 if (batch.complete(response.baseOffset, response.logAppendTime)) {
 maybeRemoveAndDeallocateBatch(batch);
+} else {
+// Always safe to call dellocate because the batch keeps track of 
whether or not it was deallocated yet
+this.accumulator.deallocateAlreadyRemovedIncomplete(batch);
 }
 }
 
 private void failBatch(ProducerBatch batch,
ProduceResponse.PartitionResponse response,
-   boolean adjustSequenceNumbers) {
+   boolean adjustSequenceNumbers,
+   boolean deallocateBatch) {

Review Comment:
   It appears the input is always `true`, so  this argument might be redundant?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19012: Fix rare producer message corruption, don't reuse buffers on the client in certain error cases [kafka]

2026-01-06 Thread via GitHub


junrao commented on code in PR #21065:
URL: https://github.com/apache/kafka/pull/21065#discussion_r2665725049


##
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java:
##
@@ -3591,6 +3596,44 @@ public void 
testWhenProduceResponseReturnsWithALeaderShipChangeErrorAndNewLeader
 }
 }
 
+@Test
+public void testNoBufferReuseWhenBatchHasTimeoutErrors() throws Exception {

Review Comment:
   HasTimeoutErrors => Expires



##
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java:
##
@@ -2499,12 +2500,16 @@ public void testNoDoubleDeallocation() throws Exception 
{
 sender.runOnce();  // send request
 assertEquals(1, client.inFlightRequestCount());
 assertEquals(1, sender.inFlightBatches(tp0).size());
+assertFalse(sender.inFlightBatches(tp0).get(0).isBufferDeallocated(), 
"Buffer not deallocated yet");
+ProducerBatch inflightBatch = sender.inFlightBatches(tp0).get(0);
+
 
 time.sleep(REQUEST_TIMEOUT);
 assertFalse(pool.allMatch());
 
 sender.runOnce();  // expire the batch
 assertTrue(request1.isDone());
+assertTrue(inflightBatch.isBufferDeallocated(), "Buffer shold be 
deallocated after request timeout");

Review Comment:
   typo shold
   
   Also, a couple of existing issues. 
   1. It seems that we unnecessarily call pool.allMatch() twice below.
   2. A couple of lines above, "expire the batch" => "times out the request"



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerBatch.java:
##
@@ -72,6 +73,7 @@ private enum FinalState { ABORTED, FAILED, SUCCEEDED }
 private final AtomicInteger attempts = new AtomicInteger(0);
 private final boolean isSplitBatch;
 private final AtomicReference finalState = new 
AtomicReference<>(null);
+private final AtomicBoolean bufferDeallocated = new AtomicBoolean(false);

Review Comment:
   Since this is only set and checked by the Sender thread, we could just use a 
regular boolean.



##
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java:
##
@@ -3591,6 +3596,44 @@ public void 
testWhenProduceResponseReturnsWithALeaderShipChangeErrorAndNewLeader
 }
 }
 
+@Test
+public void testNoBufferReuseWhenBatchHasTimeoutErrors() throws Exception {
+long totalSize = 1024 * 1024;
+try (Metrics m = new Metrics()) {
+BufferPool pool = new BufferPool(totalSize, batchSize, m, time, 
"producer-internal-metrics");
+
+// Allocate and store a poolable buffer, then return it to the 
pool so the Sender can pick it up
+ByteBuffer buffer = pool.allocate(batchSize, 0);
+pool.deallocate(buffer);
+
+setupWithTransactionState(null, false, pool);
+StringBuilder sb = new StringBuilder();
+// Make a value the right size so that RecordAccumulator calls 
BufferPool.allocate with the batchSize so it's pooled
+for (int i = 0; i < 16290; i++) {
+sb.append("a");
+}

Review Comment:
   The above could just be 
   
   `String largeValue = "a".repeat(16290);`
   
   Also, why is it necessary to create a large value? RecordAccumulator picks 
batchSize for allocation if the data size is smaller.
   
   ```
   int size = Math.max(this.batchSize, 
AbstractRecords.estimateSizeInBytesUpperBound(
   RecordBatch.CURRENT_MAGIC_VALUE, 
compression.type(), key, value, headers));
   log.trace("Allocating a new {} byte message buffer for 
topic {} partition {} with remaining timeout {}ms", size, topic, 
effectivePartition, maxTimeToBlock);
   // This call may block if we exhausted buffer space.
   buffer = free.allocate(size, maxTimeToBlock);
   ```



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1031,12 +1031,30 @@ BuiltInPartitioner createBuiltInPartitioner(LogContext 
logContext, String topic,
  */
 public void deallocate(ProducerBatch batch) {
 incomplete.remove(batch);
+deallocateIfNecessary(batch);
+}
+
+private void deallocateIfNecessary(ProducerBatch batch) {
 // Only deallocate the batch if it is not a split batch because split 
batch are allocated outside the
 // buffer pool.
-if (!batch.isSplitBatch())
+if (!batch.isSplitBatch() && !batch.isBufferDeallocated() && 
batch.setBufferDeallocated(true))

Review Comment:
   Could we log a warn if this method is called but 
`batch.isBufferDeallocated()` is true?



##
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##
@@ -1031,12 +1031,30 @@ BuiltInPartitioner createBuiltInPartitioner(LogContext 
logContext, String topic,
  */