kevin-wu24 opened a new pull request, #23737:
URL: https://github.com/apache/kafka/pull/23737

   ### Issue Description
   When the metadata layer (e.g. `MetadataLoader`) tries to read a slice of 
records from KRaft which is greater than 
`KafkaRaftClient#MAX_BATCH_SIZE_BYTES`, the `RecordsIterator` can return 
records which are past the HWM because `FileRecords#readInto` reads to the 
minimum of the end of the file and the remaining space in the associated 
buffer. In the case of the former, this can lead to a metadata error since 
uncommitted data becomes visible to the metadata layer.
   
   ### What Changed
   Add `FileRecords#readUntil` which reads from the underlying file to the 
minimum of:
   
   - the remaining space in the associated buffer
   - the end byte position of the `FileRecords` object, rather than the end of 
the underlying file. This ensures when a caller constructs a `FileRecords` 
slice of [start position, HWM), no bytes past the HWM are read into memory.
   
   ### Testing
   Added unit tests in `RecordsIteratorTest` and `FileRecordsTest`
   
   ### Rejected Alternatives
   
   - Limiting the buffer in `RecordsIterator` prior to calling 
`FileRecords#readInto`. The motivation behind this is that other callers may 
want to this functionality in the future, so the change belongs in 
`FileRecords`, not a caller of that class
   - Changing the implementation of `FileRecords#readInto`. The main concern I 
had with this was that there are many other callers of this method outside of 
the raft layer, and it is not clear to me the impact of this change on them. 
Adding an additional method does not impact existing behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to