kevin-wu24 opened a new pull request, #23737: URL: https://github.com/apache/kafka/pull/23737
### Issue Description When the metadata layer (e.g. `MetadataLoader`) tries to read a slice of records from KRaft which is greater than `KafkaRaftClient#MAX_BATCH_SIZE_BYTES`, the `RecordsIterator` can return records which are past the HWM because `FileRecords#readInto` reads to the minimum of the end of the file and the remaining space in the associated buffer. In the case of the former, this can lead to a metadata error since uncommitted data becomes visible to the metadata layer. ### What Changed Add `FileRecords#readUntil` which reads from the underlying file to the minimum of: - the remaining space in the associated buffer - the end byte position of the `FileRecords` object, rather than the end of the underlying file. This ensures when a caller constructs a `FileRecords` slice of [start position, HWM), no bytes past the HWM are read into memory. ### Testing Added unit tests in `RecordsIteratorTest` and `FileRecordsTest` ### Rejected Alternatives - Limiting the buffer in `RecordsIterator` prior to calling `FileRecords#readInto`. The motivation behind this is that other callers may want to this functionality in the future, so the change belongs in `FileRecords`, not a caller of that class - Changing the implementation of `FileRecords#readInto`. The main concern I had with this was that there are many other callers of this method outside of the raft layer, and it is not clear to me the impact of this change on them. Adding an additional method does not impact existing behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
