junrao commented on code in PR #23737:
URL: https://github.com/apache/kafka/pull/23737#discussion_r4274089077
##########
raft/src/main/java/org/apache/kafka/raft/internals/RecordsIterator.java:
##########
@@ -136,9 +136,11 @@ private MemoryRecords createMemoryRecords(FileRecords
fileRecords) {
MemoryRecords memoryRecords = readFileRecords(fileRecords, buffer);
- // firstBatchSize() is always non-null because the minimum buffer is
HEADER_SIZE_UP_TO_MAGIC.
- int firstBatchSize = memoryRecords.firstBatchSize();
- if (firstBatchSize <= buffer.remaining()) {
+ Integer firstBatchSize = memoryRecords.firstBatchSize();
+ int bytesAvailable = buffer.remaining() + (fileRecords.sizeInBytes() -
bytesRead);
+ if (firstBatchSize == null || firstBatchSize <= buffer.remaining() ||
+ firstBatchSize > bytesAvailable
Review Comment:
Here is an example. Suppose that the batch size is 1MB. The slice contains
record batch of 8MB but has a length of only 3MB. createMemoryRecords() is
called, readFileRecords() fills memoryRecords with 1MB from the slice. It then
notices firstBatchSize (8MB) > bytesAvailable (3MB) and returns memoryRecords
without growing the buffer. The caller will see no records in the returned
memoryRecords but with bytesRead < recordSize. createMemoryRecords() is called
again. Since memoryRecords is full, readFileRecords() reads nothing.
firstBatchSize (8MB) > bytesAvailable (3MB) is still true and the same
memoryRecords is returned. This will loop forever.
Note that the above won't happen without the optimization since it will grow
the buffer and drain all bytes of the partial batch from the records.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]