kevin-wu24 commented on code in PR #23737:
URL: https://github.com/apache/kafka/pull/23737#discussion_r4329966752
##########
raft/src/main/java/org/apache/kafka/raft/internals/RecordsIterator.java:
##########
@@ -136,18 +136,23 @@ private MemoryRecords createMemoryRecords(FileRecords
fileRecords) {
MemoryRecords memoryRecords = readFileRecords(fileRecords, buffer);
- // firstBatchSize() is always non-null because the minimum buffer is
HEADER_SIZE_UP_TO_MAGIC.
- int firstBatchSize = memoryRecords.firstBatchSize();
- if (firstBatchSize <= buffer.remaining()) {
+ Integer firstBatchSize = memoryRecords.firstBatchSize();
+ if (firstBatchSize != null && firstBatchSize <= buffer.remaining()) {
+ // The buffer contains at least one complete batch
return memoryRecords;
+ } else if (firstBatchSize == null || bytesRead >=
fileRecords.sizeInBytes()) {
+ // No more bytes can be read, but the buffer doesn't contain a
complete batch. firstBatchSize
+ // is null when fewer than HEADER_SIZE_UP_TO_MAGIC bytes are left.
Callers are expected to
+ // pass records that end on a batch boundary, so reaching this
code is unexpected.
+ throw new IllegalStateException("Unexpected partial batch of
records");
Review Comment:
Discussed this issue offline with @jsancio. The main takeaways were:
- Enforcing batch alignment of records at the `RecordsIterator` level via
throwing an exception is mainly motivated by assumptions around the current
caller behaviors, which are raft/metadata module codes. If we want to enforce
this without assuming things about the callers of this iterator, batch
alignment should instead be enforced at the underlying records level. Meaning,
if the underlying`FileRecords` type does not currently enforce batch alignment
of the end offset in either the slice or the mutable case, why should
`RecordsIterator`? If the raft module wants to enforce this, perhaps it can use
some type like `BatchAlignedBoundedRecords`, which itself enforces the records
are an immutable slice, and are batch aligned. This way, the `RecordsIterator`
type can remain generic and reusable. The case specifically mentioned was that
the one caller of `RecordsIterator` on uncommitted data in
`KRaftControlRecordStateMachine` could have an implementation that holds the
same `RecordsIt
erator` object and waits for new bytes (which may not be batch aligned) to
come in.
- Allowing for a partial batch at the end of the requested slice is okay
because `memoryRecords.batchIterator()#hasNext() == false`, and so will the
actual `RecordsIterator#hasNext() == false`. This means the overall Iterator
contract is not violated. Meaning, a partial batch at the end of a
`FileRecords` slice do not make it impossible for `hasNext()` to return false,
so this is not an invalid state.
I am going to revert the exception code and tests because of the above. What
do you think @junrao?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]