junrao commented on code in PR #23737:
URL: https://github.com/apache/kafka/pull/23737#discussion_r4318808370
##########
raft/src/test/java/org/apache/kafka/raft/internals/RecordsIteratorTest.java:
##########
@@ -189,14 +189,11 @@ public void testFileRecordsSliceEndingInPartialBatch(int
partialBatchBytes) thro
firstBatchSize,
true
)) {
- iterator.forEachRemaining(batch ->
actualBatches.add(TestBatch.from(batch)));
+ assertThrows(
+ IllegalStateException.class,
+ () -> iterator.forEachRemaining(batch ->
actualBatches.add(TestBatch.from(batch)))
Review Comment:
Claude suggested the following.
```
2. testFileRecordsSliceEndingInPartialBatch doesn't check what it sets up.
It builds actualBatches and allocatedBuffers but asserts on neither. After the
try block, add:
assertEquals(List.of(firstBatch), actualBatches); // the complete batch is
still returned before the throw
assertEquals(Set.of(), allocatedBuffers); // buffers are released
on the exception path
```
##########
raft/src/main/java/org/apache/kafka/raft/internals/RecordsIterator.java:
##########
@@ -136,18 +136,23 @@ private MemoryRecords createMemoryRecords(FileRecords
fileRecords) {
MemoryRecords memoryRecords = readFileRecords(fileRecords, buffer);
- // firstBatchSize() is always non-null because the minimum buffer is
HEADER_SIZE_UP_TO_MAGIC.
- int firstBatchSize = memoryRecords.firstBatchSize();
- if (firstBatchSize <= buffer.remaining()) {
+ Integer firstBatchSize = memoryRecords.firstBatchSize();
+ if (firstBatchSize != null && firstBatchSize <= buffer.remaining()) {
+ // The buffer contains at least one complete batch
return memoryRecords;
+ } else if (firstBatchSize == null || bytesRead >=
fileRecords.sizeInBytes()) {
+ // No more bytes can be read, but the buffer doesn't contain a
complete batch. firstBatchSize
+ // is null when fewer than HEADER_SIZE_UP_TO_MAGIC bytes are left.
Callers are expected to
+ // pass records that end on a batch boundary, so reaching this
code is unexpected.
+ throw new IllegalStateException("Unexpected partial batch of
records");
Review Comment:
Claude suggests the following.
```
1. An incomplete batch on the grow path still ends iteration without an
error (RecordsIterator.java:~143-161). The new branch only throws once
everything has been read (bytesRead >= size). If the slice still has unread
bytes but the declared batch is bigger than what's left, the code grows the
buffer, reads the rest, and returns records with no complete batch. hasNext()
is then false, with no exception. The reviewer's example:
batchSize = 17 (the default in createIterator), and a slice holding the
first 30 bytes of a 100-byte batch.
Read 1 gets 17 bytes, firstBatchSize = 100, and bytesRead = 17 < 30.
The buffer grows to 100 bytes, read 2 gets 13 bytes, and iteration ends with
nothing returned and no error.
```
```
3. Say what went wrong in the exception message. If this fires while loading
a KRaft snapshot or a committed slice, "Unexpected partial batch of records"
says nothing about where. Including details costs nothing:
throw new IllegalStateException(String.format(
"Records %s end in a partial batch: %d bytes buffered at position %d,
batch size %s, %d of %d bytes read",
fileRecords, buffer.remaining(), bytesRead - buffer.remaining(),
firstBatchSize == null ? "unknown" : firstBatchSize, bytesRead,
fileRecords.sizeInBytes()));
bytesRead
```
RecordsIterator supports both MemoryRecords and FileRecords. MemoryRecords
input still ignores a trailing incomplete batch through batchIterator(), while
FileRecords now throws. Could we make the behavior consistent and also document
this in the javadoc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]