junrao commented on code in PR #23737:
URL: https://github.com/apache/kafka/pull/23737#discussion_r4318808370


##########
raft/src/test/java/org/apache/kafka/raft/internals/RecordsIteratorTest.java:
##########
@@ -189,14 +189,11 @@ public void testFileRecordsSliceEndingInPartialBatch(int 
partialBatchBytes) thro
                 firstBatchSize,
                 true
             )) {
-                iterator.forEachRemaining(batch -> 
actualBatches.add(TestBatch.from(batch)));
+                assertThrows(
+                    IllegalStateException.class,
+                    () -> iterator.forEachRemaining(batch -> 
actualBatches.add(TestBatch.from(batch)))

Review Comment:
   Claude suggested the following.
   
   ```
   2. testFileRecordsSliceEndingInPartialBatch doesn't check what it sets up. 
It builds actualBatches and allocatedBuffers but asserts on neither. After the 
try block, add:
   
   
   
   assertEquals(List.of(firstBatch), actualBatches);   // the complete batch is 
still returned before the throw
   assertEquals(Set.of(), allocatedBuffers);           // buffers are released 
on the exception path
   ```



##########
raft/src/main/java/org/apache/kafka/raft/internals/RecordsIterator.java:
##########
@@ -136,18 +136,23 @@ private MemoryRecords createMemoryRecords(FileRecords 
fileRecords) {
 
         MemoryRecords memoryRecords = readFileRecords(fileRecords, buffer);
 
-        // firstBatchSize() is always non-null because the minimum buffer is 
HEADER_SIZE_UP_TO_MAGIC.
-        int firstBatchSize = memoryRecords.firstBatchSize();
-        if (firstBatchSize <= buffer.remaining()) {
+        Integer firstBatchSize = memoryRecords.firstBatchSize();
+        if (firstBatchSize != null && firstBatchSize <= buffer.remaining()) {
+            // The buffer contains at least one complete batch
             return memoryRecords;
+        } else if (firstBatchSize == null || bytesRead >= 
fileRecords.sizeInBytes()) {
+            // No more bytes can be read, but the buffer doesn't contain a 
complete batch. firstBatchSize
+            // is null when fewer than HEADER_SIZE_UP_TO_MAGIC bytes are left. 
Callers are expected to
+            // pass records that end on a batch boundary, so reaching this 
code is unexpected.
+            throw new IllegalStateException("Unexpected partial batch of 
records");

Review Comment:
   Claude suggests the following.
   
   ```
   1. An incomplete batch on the grow path still ends iteration without an 
error (RecordsIterator.java:~143-161). The new branch only throws once 
everything has been read (bytesRead >= size). If the slice still has unread 
bytes but the declared batch is bigger than what's left, the code grows the 
buffer, reads the rest, and returns records with no complete batch. hasNext() 
is then false, with no exception. The reviewer's example:
   
   batchSize = 17 (the default in createIterator), and a slice holding the 
first 30 bytes of a 100-byte batch.
   
   Read 1 gets 17 bytes, firstBatchSize = 100, and bytesRead = 17 < 30.
   The buffer grows to 100 bytes, read 2 gets 13 bytes, and iteration ends with 
nothing returned and no error.
   ```
   
   ```
   3. Say what went wrong in the exception message. If this fires while loading 
a KRaft snapshot or a committed slice, "Unexpected partial batch of records" 
says nothing about where. Including details costs nothing:
   
   throw new IllegalStateException(String.format(
       "Records %s end in a partial batch: %d bytes buffered at position %d, 
batch size %s, %d of %d bytes read",
       fileRecords, buffer.remaining(), bytesRead - buffer.remaining(),
       firstBatchSize == null ? "unknown" : firstBatchSize, bytesRead, 
fileRecords.sizeInBytes()));
   bytesRead
   ```
   
   RecordsIterator supports both MemoryRecords and FileRecords. MemoryRecords 
input still ignores a trailing incomplete batch through batchIterator(), while 
FileRecords now throws. Could we make the behavior consistent and also document 
this in the javadoc?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to