junrao commented on code in PR #23737:
URL: https://github.com/apache/kafka/pull/23737#discussion_r4261352139
##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -128,15 +126,19 @@ public FileChannel channel() {
}
/**
- * Read log batches into the given buffer until there are no bytes
remaining in the buffer or the end of the file
- * is reached.
+ * Read log batches into the given buffer until there are no bytes
remaining in the buffer or the end of
+ * this FileRecords object is reached.
*
* @param buffer The buffer to write the batches to
- * @param position Position in the buffer to read from
+ * @param position Position to start reading from, relative to the start
of this FileRecords object
* @throws IOException If an I/O error occurs, see {@link
FileChannel#read(ByteBuffer, long)} for details on the
* possible exceptions
*/
public void readInto(ByteBuffer buffer, int position) throws IOException {
+ int size = sizeInBytes();
Review Comment:
Perhaps rename size to sth like currentSize to avoid hiding the size field.
##########
clients/src/test/java/org/apache/kafka/common/record/internal/FileRecordsTest.java:
##########
@@ -240,6 +241,73 @@ public void testRead() {
assertEquals(Collections.singletonList(second), batches(read), "Read a
single message starting from the second message");
}
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ public void testReadIntoStopsAtEndOfSlice(boolean preallocate) throws
IOException {
+ try (FileRecords fileRecords = createFileRecords(values, preallocate))
{
+ List<RecordBatch> items = batches(fileRecords);
+ RecordBatch second = items.get(1);
+ // The slice holds only the second batch. The third batch should
not be read into the buffer.
+ FileRecords slice = fileRecords.slice(items.get(0).sizeInBytes(),
second.sizeInBytes());
+ ByteBuffer buffer = ByteBuffer.allocate(fileRecords.sizeInBytes());
+
+ slice.readInto(buffer, 0);
+
+ assertEquals(second.sizeInBytes(), buffer.remaining());
+ assertEquals(
+ List.of(second.baseOffset()),
+ batches(MemoryRecords.readableRecords(buffer)).stream().
+ map(RecordBatch::baseOffset).collect(Collectors.toList())
+ );
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
Review Comment:
Claude suggested the following.
`preallocate parameter: only testReadIntoStopsAtEndOfRecords needs it. The
other three tests read only written data, so the second run adds nothing.`
##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -128,15 +126,19 @@ public FileChannel channel() {
}
/**
- * Read log batches into the given buffer until there are no bytes
remaining in the buffer or the end of the file
- * is reached.
+ * Read log batches into the given buffer until there are no bytes
remaining in the buffer or the end of
+ * this FileRecords object is reached.
*
* @param buffer The buffer to write the batches to
- * @param position Position in the buffer to read from
+ * @param position Position to start reading from, relative to the start
of this FileRecords object
Review Comment:
Was this comment addressed?
##########
raft/src/main/java/org/apache/kafka/raft/internals/RecordsIterator.java:
##########
@@ -136,9 +136,10 @@ private MemoryRecords createMemoryRecords(FileRecords
fileRecords) {
MemoryRecords memoryRecords = readFileRecords(fileRecords, buffer);
- // firstBatchSize() is always non-null because the minimum buffer is
HEADER_SIZE_UP_TO_MAGIC.
- int firstBatchSize = memoryRecords.firstBatchSize();
- if (firstBatchSize <= buffer.remaining()) {
+ Integer firstBatchSize = memoryRecords.firstBatchSize();
+ if (firstBatchSize == null || firstBatchSize <= buffer.remaining() ||
+ bytesRead >= fileRecords.sizeInBytes()
Review Comment:
Claude suggested the following improvement.
```
1. Grow the buffer only when the batch can actually be completed
(RecordsIterator.java:140-151). Today the grow path runs whenever the first
batch doesn't fit in the buffer and the slice isn't fully read. The buffer is
then sized to firstBatchSize, even when that batch runs past the end of the
slice. That has two costs:
It allocates for a batch that can never be completed.
A corrupt length field (firstBatchSize() parses with maxMessageSize =
Integer.MAX_VALUE) can trigger a very large allocation, e.g. an
OutOfMemoryError on the controller while it reads a snapshot. The capped read
would then fill only a small part of that buffer.
The old code had the same problem, but it's easy to fix here. Compare
against the bytes the slice can still supply; that also replaces the bytesRead
>= size check:
Integer firstBatchSize = memoryRecords.firstBatchSize();
int bytesAvailable = buffer.remaining() + (fileRecords.sizeInBytes() -
bytesRead);
if (firstBatchSize == null || firstBatchSize <= buffer.remaining() ||
firstBatchSize > bytesAvailable) {
return memoryRecords;
}
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]