jsancio commented on code in PR #23737:
URL: https://github.com/apache/kafka/pull/23737#discussion_r4327614303
##########
clients/src/main/java/org/apache/kafka/common/record/internal/FileRecords.java:
##########
@@ -128,15 +126,20 @@ public FileChannel channel() {
}
/**
- * Read log batches into the given buffer until there are no bytes
remaining in the buffer or the end of the file
- * is reached.
+ * Read log batches into the given buffer until there are no bytes
remaining in the buffer or the end of
+ * this FileRecords object is reached.
*
* @param buffer The buffer to write the batches to
- * @param position Position in the buffer to read from
+ * @param position Position to start reading from, relative to the start
of this FileRecords object
* @throws IOException If an I/O error occurs, see {@link
FileChannel#read(ByteBuffer, long)} for details on the
- * possible exceptions
+ * possible exceptions.
+ * @throws IllegalArgumentException If position is negative or greater
than {@link #sizeInBytes()}
*/
public void readInto(ByteBuffer buffer, int position) throws IOException {
+ int currentSize = sizeInBytes();
+ if (position < 0 || position > currentSize)
+ throw new IllegalArgumentException("Read position " + position + "
is outside " + this);
Review Comment:
Can we enclose this statement using `{ }`:
```java
if (position < 0 || position > currentSize) {
throw new IllegalArgumentException("Read position " + position +
" is outside " + this);
}
```
##########
raft/src/main/java/org/apache/kafka/raft/internals/RecordsIterator.java:
##########
@@ -136,18 +136,23 @@ private MemoryRecords createMemoryRecords(FileRecords
fileRecords) {
MemoryRecords memoryRecords = readFileRecords(fileRecords, buffer);
- // firstBatchSize() is always non-null because the minimum buffer is
HEADER_SIZE_UP_TO_MAGIC.
- int firstBatchSize = memoryRecords.firstBatchSize();
- if (firstBatchSize <= buffer.remaining()) {
+ Integer firstBatchSize = memoryRecords.firstBatchSize();
+ if (firstBatchSize != null && firstBatchSize <= buffer.remaining()) {
+ // The buffer contains at least one complete batch
return memoryRecords;
+ } else if (firstBatchSize == null || bytesRead >=
fileRecords.sizeInBytes()) {
+ // No more bytes can be read, but the buffer doesn't contain a
complete batch. firstBatchSize
+ // is null when fewer than HEADER_SIZE_UP_TO_MAGIC bytes are left.
Callers are expected to
+ // pass records that end on a batch boundary, so reaching this
code is unexpected.
+ throw new IllegalStateException("Unexpected partial batch of
records");
Review Comment:
I am not sure that we should throw an exception if FileRecords doesn't have
enough bytes to complete a batch. This can happen when concurrently reading
uncommitted data to the LEO and another thread appends to the LEO. Or when the
slice doesn't end in a batch boundary.
I think we need to distinguish the two cases:
1. The buffer is not big enough to read the entire batch - create a new
batch that is big enough to read the batch from the file records.
2. The file record (slice or not) doesn't contain an entire batch - return
to the user that there are no more batches.
I guess the issue is that if there are concurrent read and write from the
LEO `RecordsIterator::hasNext` may return false in one call to later return
true. Is this what we are trying to fix? I took a look at the Java
documentation for Iterator and the documentation does forbid this. I would
assume that concurrent iterator need to be able to return false in one call and
true in a future call.
What do we think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]