junrao commented on code in PR #23737:
URL: https://github.com/apache/kafka/pull/23737#discussion_r4261352139


##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -128,15 +126,19 @@ public FileChannel channel() {
     }
 
     /**
-     * Read log batches into the given buffer until there are no bytes 
remaining in the buffer or the end of the file
-     * is reached.
+     * Read log batches into the given buffer until there are no bytes 
remaining in the buffer or the end of
+     * this FileRecords object is reached.
      *
      * @param buffer The buffer to write the batches to
-     * @param position Position in the buffer to read from
+     * @param position Position to start reading from, relative to the start 
of this FileRecords object
      * @throws IOException If an I/O error occurs, see {@link 
FileChannel#read(ByteBuffer, long)} for details on the
      * possible exceptions
      */
     public void readInto(ByteBuffer buffer, int position) throws IOException {
+        int size = sizeInBytes();

Review Comment:
   Perhaps rename size to sth like currentSize to avoid hiding the size field.



##########
clients/src/test/java/org/apache/kafka/common/record/internal/FileRecordsTest.java:
##########
@@ -240,6 +241,73 @@ public void testRead() {
         assertEquals(Collections.singletonList(second), batches(read), "Read a 
single message starting from the second message");
     }
 
+    @ParameterizedTest
+    @ValueSource(booleans = {true, false})
+    public void testReadIntoStopsAtEndOfSlice(boolean preallocate) throws 
IOException {
+        try (FileRecords fileRecords = createFileRecords(values, preallocate)) 
{
+            List<RecordBatch> items = batches(fileRecords);
+            RecordBatch second = items.get(1);
+            // The slice holds only the second batch. The third batch should 
not be read into the buffer.
+            FileRecords slice = fileRecords.slice(items.get(0).sizeInBytes(), 
second.sizeInBytes());
+            ByteBuffer buffer = ByteBuffer.allocate(fileRecords.sizeInBytes());
+
+            slice.readInto(buffer, 0);
+
+            assertEquals(second.sizeInBytes(), buffer.remaining());
+            assertEquals(
+                List.of(second.baseOffset()),
+                batches(MemoryRecords.readableRecords(buffer)).stream().
+                    map(RecordBatch::baseOffset).collect(Collectors.toList())
+            );
+        }
+    }
+
+    @ParameterizedTest
+    @ValueSource(booleans = {true, false})

Review Comment:
   Claude suggested the following.
   
   `preallocate parameter: only testReadIntoStopsAtEndOfRecords needs it. The 
other three tests read only written data, so the second run adds nothing.`



##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -128,15 +126,19 @@ public FileChannel channel() {
     }
 
     /**
-     * Read log batches into the given buffer until there are no bytes 
remaining in the buffer or the end of the file
-     * is reached.
+     * Read log batches into the given buffer until there are no bytes 
remaining in the buffer or the end of
+     * this FileRecords object is reached.
      *
      * @param buffer The buffer to write the batches to
-     * @param position Position in the buffer to read from
+     * @param position Position to start reading from, relative to the start 
of this FileRecords object

Review Comment:
   Was this comment addressed?



##########
raft/src/main/java/org/apache/kafka/raft/internals/RecordsIterator.java:
##########
@@ -136,9 +136,10 @@ private MemoryRecords createMemoryRecords(FileRecords 
fileRecords) {
 
         MemoryRecords memoryRecords = readFileRecords(fileRecords, buffer);
 
-        // firstBatchSize() is always non-null because the minimum buffer is 
HEADER_SIZE_UP_TO_MAGIC.
-        int firstBatchSize = memoryRecords.firstBatchSize();
-        if (firstBatchSize <= buffer.remaining()) {
+        Integer firstBatchSize = memoryRecords.firstBatchSize();
+        if (firstBatchSize == null || firstBatchSize <= buffer.remaining() ||
+            bytesRead >= fileRecords.sizeInBytes()

Review Comment:
   Claude suggested the following improvement.
   
   ```
   1. Grow the buffer only when the batch can actually be completed 
(RecordsIterator.java:140-151). Today the grow path runs whenever the first 
batch doesn't fit in the buffer and the slice isn't fully read. The buffer is 
then sized to firstBatchSize, even when that batch runs past the end of the 
slice. That has two costs:
   
   It allocates for a batch that can never be completed.
   A corrupt length field (firstBatchSize() parses with maxMessageSize = 
Integer.MAX_VALUE) can trigger a very large allocation, e.g. an 
OutOfMemoryError on the controller while it reads a snapshot. The capped read 
would then fill only a small part of that buffer.
   The old code had the same problem, but it's easy to fix here. Compare 
against the bytes the slice can still supply; that also replaces the bytesRead 
>= size check:
   
   
   
   Integer firstBatchSize = memoryRecords.firstBatchSize();
   int bytesAvailable = buffer.remaining() + (fileRecords.sizeInBytes() - 
bytesRead);
   if (firstBatchSize == null || firstBatchSize <= buffer.remaining() || 
firstBatchSize > bytesAvailable) {
       return memoryRecords;
   }
   ...
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to