junrao commented on code in PR #23737:
URL: https://github.com/apache/kafka/pull/23737#discussion_r4274089077


##########
raft/src/main/java/org/apache/kafka/raft/internals/RecordsIterator.java:
##########
@@ -136,9 +136,11 @@ private MemoryRecords createMemoryRecords(FileRecords 
fileRecords) {
 
         MemoryRecords memoryRecords = readFileRecords(fileRecords, buffer);
 
-        // firstBatchSize() is always non-null because the minimum buffer is 
HEADER_SIZE_UP_TO_MAGIC.
-        int firstBatchSize = memoryRecords.firstBatchSize();
-        if (firstBatchSize <= buffer.remaining()) {
+        Integer firstBatchSize = memoryRecords.firstBatchSize();
+        int bytesAvailable = buffer.remaining() + (fileRecords.sizeInBytes() - 
bytesRead);
+        if (firstBatchSize == null || firstBatchSize <= buffer.remaining() ||
+            firstBatchSize > bytesAvailable

Review Comment:
   Here is an example. Suppose that the batch size is 1MB. The slice contains 
record batch of 8MB but has a length of only 3MB. createMemoryRecords() is 
called, readFileRecords() fills memoryRecords with 1MB from the slice. It then 
notices firstBatchSize (8MB) > bytesAvailable (3MB) and returns memoryRecords 
without growing the buffer. The caller will see no records in the returned 
memoryRecords but with bytesRead < recordSize. createMemoryRecords() is called 
again. Since memoryRecords is full,  readFileRecords() reads nothing. 
firstBatchSize (8MB) > bytesAvailable (3MB) is still true and the same 
memoryRecords is returned. This will loop  forever.
   
   Note that the above won't happen without the optimization since it will grow 
the buffer and drain all bytes of the partial batch from the records.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to