kevin-wu24 commented on code in PR #23737:
URL: https://github.com/apache/kafka/pull/23737#discussion_r4272544405
##########
raft/src/main/java/org/apache/kafka/raft/internals/RecordsIterator.java:
##########
@@ -136,9 +136,11 @@ private MemoryRecords createMemoryRecords(FileRecords
fileRecords) {
MemoryRecords memoryRecords = readFileRecords(fileRecords, buffer);
- // firstBatchSize() is always non-null because the minimum buffer is
HEADER_SIZE_UP_TO_MAGIC.
- int firstBatchSize = memoryRecords.firstBatchSize();
- if (firstBatchSize <= buffer.remaining()) {
+ Integer firstBatchSize = memoryRecords.firstBatchSize();
+ int bytesAvailable = buffer.remaining() + (fileRecords.sizeInBytes() -
bytesRead);
+ if (firstBatchSize == null || firstBatchSize <= buffer.remaining() ||
+ firstBatchSize > bytesAvailable
Review Comment:
> It means that bytesRead may never reach records.sizeInBytes(), but
nextBatches() depends on that to end the iteration.
I'm not sure if I agree with this statement. I think in
`RecordsIterator#nextBatch()`, `RecordsIterator#nextBatches =
memoryRecords.batchIterator()` with `RecordBatchIterator#hasNext() == false` in
the `firstBatchSize == null` or `firstBatchSize > bytesAvailable` case, because
there is not a full batch in the buffer. I looked at
`ByteBufferLogInputStream#nextBatch()`, `RecordBatchIterator#makeNext()`, and
`AbstractIterator`. Because `nextBatches.hasNext() == false`,
`RecordsIterator#nextBatch()` returns an empty optional, so
`RecordsIterator#hasNext` will return false eventually. Any code that calls
`RecordsIterator#hasNext` again after it evaluates false for the first time is
wrong, and I am pretty sure that does not happen in the `raft` module.
> Currently, it doesn't seem to cause an issue, but it makes the code harder
to understand
I believe the reason for this is that it is the caller's (raft code's)
responsibility that this iterator is requested at batch aligned offsets. This
should always be the case because the HWM/LEO are batch aligned, but I don't
think the code enforces that in any way.
I do agree this is harder to understand than the `bytesRead >=
fileRecords.sizeInBytes()` condition. I think case 1 and 3 (firstBatchSize >
bytesAvailable) of this if clause should only ever be hit if the user of this
iterator calls it on a records slice which is not batch aligned, which I would
argue is unexpected. In that case, should the code do something stronger: (e.g.
log an error, or maybe throw)? I am not sure about throwing because that can
impact availability.
What do you think? cc @jsancio @junrao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]