lucasfang commented on code in PR #513:
URL: https://github.com/apache/paimon-cpp/pull/513#discussion_r4545114982
##########
src/paimon/format/orc/orc_file_batch_reader.cpp:
##########
@@ -211,10 +222,20 @@ Status OrcFileBatchReader::SeekToRow(uint64_t row_number)
{
}
Result<std::vector<std::pair<uint64_t, uint64_t>>>
OrcFileBatchReader::PreBufferRange() {
- return reader_->PreBufferRange(target_column_ids_);
+ using ByteRanges = std::vector<std::pair<uint64_t, uint64_t>>;
+ std::pair<uint64_t, uint64_t> row_range;
+ PAIMON_ASSIGN_OR_RAISE(ByteRanges ranges,
+ reader_->PreBufferRange(target_column_ids_,
&row_range));
+ pre_buffered_row_range_ = row_range;
+ return ranges;
}
Result<BatchReader::ReadBatch> OrcFileBatchReader::NextBatch() {
+ uint64_t next_row = reader_->GetNextRowToRead();
+ if (read_ahead_ && (!pre_buffered_row_range_ || next_row <
pre_buffered_row_range_->first ||
+ next_row >= pre_buffered_row_range_->second)) {
+ PAIMON_RETURN_NOT_OK(read_ahead_->PrepareRanges(this));
Review Comment:
Thanks for the detailed reproduction. You're right about both the seek
ordering and the test's inability to prove that decoding consumes the cached
bytes.
In 5a70ca51, I removed the ORC per-stripe warmup optimization and its
dedicated tests from this PR to keep the scope focused on decoupling RAW warmup
and read-ahead caching from batch prefetch. ORC now retains its existing
all-stripe `PreBufferRange()` implementation, with no new range-preparation
hooks in `NextBatch()` or `SeekToRow()`. This removes the late per-stripe
preparation path identified here. The existing cost of discovering all stripe
ranges is documented; any future per-stripe optimization should cover
seek-triggered I/O and verify physical reads/cache hits as you suggested.
The real-ORC decoupling tests remain, covering synchronous reads, parallel
prefetch, adaptive fallback, and late materialization. The five affected test
targets passed locally (5,228 passed, 2 conditional skips).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]