kevin-wu24 commented on code in PR #23737:
URL: https://github.com/apache/kafka/pull/23737#discussion_r4329966752


##########
raft/src/main/java/org/apache/kafka/raft/internals/RecordsIterator.java:
##########
@@ -136,18 +136,23 @@ private MemoryRecords createMemoryRecords(FileRecords 
fileRecords) {
 
         MemoryRecords memoryRecords = readFileRecords(fileRecords, buffer);
 
-        // firstBatchSize() is always non-null because the minimum buffer is 
HEADER_SIZE_UP_TO_MAGIC.
-        int firstBatchSize = memoryRecords.firstBatchSize();
-        if (firstBatchSize <= buffer.remaining()) {
+        Integer firstBatchSize = memoryRecords.firstBatchSize();
+        if (firstBatchSize != null && firstBatchSize <= buffer.remaining()) {
+            // The buffer contains at least one complete batch
             return memoryRecords;
+        } else if (firstBatchSize == null || bytesRead >= 
fileRecords.sizeInBytes()) {
+            // No more bytes can be read, but the buffer doesn't contain a 
complete batch. firstBatchSize
+            // is null when fewer than HEADER_SIZE_UP_TO_MAGIC bytes are left. 
Callers are expected to
+            // pass records that end on a batch boundary, so reaching this 
code is unexpected.
+            throw new IllegalStateException("Unexpected partial batch of 
records");

Review Comment:
   Discussed this issue offline with @jsancio. The main takeaways were:
   
   - Enforcing batch alignment of records at the `RecordsIterator` level via 
throwing an exception is mainly motivated by assumptions around the current 
caller behaviors, which are raft/metadata module codes. If we want to enforce 
this without assuming things about the callers of this iterator, batch 
alignment should instead be enforced at the underlying records level. Meaning, 
if the underlying`FileRecords` type does not currently enforce batch alignment 
of the end offset in either the slice or the mutable case, why should 
`RecordsIterator`? If the raft module wants to enforce this, perhaps it can use 
some type like `BatchAlignedBoundedRecords`, which itself enforces the records 
are an immutable slice, and are batch aligned. This way, the `RecordsIterator` 
type can remain generic and reusable. The case specifically mentioned was that 
the one caller of `RecordsIterator` on uncommitted data in 
`KRaftControlRecordStateMachine` could have an implementation that holds the 
same `RecordsIt
 erator` object and waits for new bytes (which may not be batch aligned) to 
come in.
   - Allowing for a partial batch at the end of the requested slice is okay 
because `memoryRecords.batchIterator()#hasNext() == false`, and so will the 
actual `RecordsIterator#hasNext() == false`. This means the overall Iterator 
contract is not violated. Meaning, a partial batch at the end of a 
`FileRecords` slice do not make it impossible for `hasNext()` to return false, 
so this is not an invalid state.
   
   I am going to revert the exception code and tests because of the above. What 
do you think @junrao?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to