zhf999 opened a new pull request, #166:
URL: https://github.com/apache/paimon-cpp/pull/166
### Purpose
Arrow caches the file-level `PageIndexReader`
(`ParquetFileReader::GetPageIndexReader()`), but
`PageIndexReader::RowGroup(i)` builds a brand-new
`RowGroupPageIndexReaderImpl` on every call, and
the OffsetIndex/ColumnIndex byte buffer (`offset_index_buffer_`) is cached
**inside that instance**.
So every new `RowGroup(i)` call re-issues a real `ReadAt` for the whole
page-index region of the row
group. These reads are not covered by the data-page pre-buffer/range cache,
so they are genuine
small I/Os against the underlying file.
Previously each stage of a page-filtered read created its own row-group
reader, so for one
partially-matched row group the page index region was read up to three times:
1. `FileReaderWrapper::CollectPreBufferRanges()` -> `ComputePageRanges()`
2. `FileReaderWrapper::NextPageFiltered()` -> `ComputePageRanges()`
3. `PageFilteredRowGroupReader::ReadFilteredRowGroup()` (for the per-field
`OffsetIndex` lookups)
This PR makes the row-group page index reader an explicit, reusable input:
- `FileReaderWrapper` gains `GetRowGroupPageIndexReader(row_group_index)`
plus a
`row_group_page_index_readers_` map that memoizes one reader per row group
(including the
"no page index" negative result), so all stages share the same page-index
buffers.
- `PageFilteredRowGroupReader::ComputePageRanges()` and
`PageFilteredRowGroupReader::ReadFilteredRowGroup()` now take the
`std::shared_ptr<::parquet::RowGroupPageIndexReader>` from the caller
instead of deriving it
internally from `ParquetFileReader::GetPageIndexReader()`.
Behavior is unchanged: files without a page index still pass a null reader
and fall back to the
existing whole-column-chunk path; read results and computed ranges are
identical. The only
difference is that the OffsetIndex region of each row group is read and
deserialized once per file
reader instead of once per stage.
### Tests
No.
### API and Format
No changes.
### Documentation
No changes.
### Generative AI tooling
Generated-by: No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
