zhf999 opened a new pull request, #167:
URL: https://github.com/apache/paimon-cpp/pull/167
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
Page-level filtering currently installs a `data_page_filter` on Arrow's
`PageReader`. This skips the page **body** (read/decrypt/decompress/decode) of
unselected pages, but `SerializedPageReader::NextPage()` still walks the column
chunk sequentially: it Peeks and Thrift-deserializes the header of **every**
page, because the only way to find the next page boundary is the
`compressed_page_size` field inside the header. On top of the page-level
`ReadRangeCache`, each skipped page header whose bytes are not cached triggers
a real positional read of up to 16 KiB (`kDefaultPageHeaderSize`), so skipping
many pages still causes significant small-read I/O amplification and
header-parsing CPU cost.
This PR eliminates header reads for unselected pages by driving the
`PageReader` with a precomputed read plan derived from the OffsetIndex:
- **Arrow patch (`cmake_modules/arrow.diff`)**
- Adds an experimental `DataPageReadPlanEntry {page_ordinal, offset,
compressed_page_size}` and
`PageReader::set_data_page_read_plan(first_data_page_offset, data_pages)`,
mutually exclusive with `set_data_page_filter`. Offsets are relative to the
column chunk stream.
- `SerializedPageReader::NextPage()` reads the region before
`first_data_page_offset` sequentially (dictionary page), then jumps directly to
each planned data page via `Advance()` and returns EOS once the plan is
exhausted. Unselected pages are never Peeked or deserialized.
- Header Peek size is clamped to `min(16 KiB, compressed_page_size)` so a
header probe never crosses the cached page range (which would fall back to real
I/O).
- **Paimon
(`src/paimon/format/parquet/page_filtered_row_group_reader.{h,cpp}`)**
- Replaces the per-page callback `MakePageFilter` with
`MakeDataPageReadPlan`, which converts RowRanges + OffsetIndex into
stream-relative plan entries.
- `ComputePageRanges` now prefetches exact page ranges `{offset,
compressed_page_size}` plus the dictionary prefix, and returns no ranges for an
empty row selection.
- An empty row selection no longer constructs a `PageReader` (the column
iterator is created with no row groups), avoiding the eager whole-column-chunk
read in `GetColumnPageReader()` when no range cache is present.
Note: rebuilding the Arrow external project is required after this change,
since `cmake_modules/arrow.diff` is applied at Arrow EP patch time and the
stamp chain does not track it automatically.
### Tests
New UT cases in
`src/paimon/format/parquet/page_filtered_row_group_reader_test.cpp`, using a
`ReadAtTrackingInputStream` that records every positional read issued while
consuming data pages and asserts that no recorded range covers an unselected
page header:
- `DirectOffsetIndexJumpDoesNotReadUnselectedPageHeaders`: selects rows from
pages 1 and 8 out of 10; asserts selected pages are read and unselected page
headers are never touched (this assertion fails before this change).
- `DirectOffsetIndexJumpReadsEachLeafDictionary`: two dictionary-encoded
leaves selecting only the last data page; each leaf still loads its own
dictionary before jumping.
- `DirectOffsetIndexJumpSupportsDictionaryFallbackToPlain`: a column that
falls back from dictionary encoding to PLAIN mid-chunk; one plan decodes both
kinds of selected pages correctly.
- `DirectOffsetIndexJumpDataPageV2AdjacentPages`: DATA_PAGE_V2 with two
adjacent selected pages, verifying the plan cursor advances exactly once per
page.
- `DictionaryEmptySelectionDoesNotReadPages`: an empty row selection reads
no dictionary/data page bytes.
- `MissingOffsetIndexFallsBackToSequentialRead`: files written without a
page index keep the original full-column read path and return correct results.
All existing page-filtering tests in this suite now exercise the direct read
plan path and continue to verify end-to-end results (flat/nested columns, V1/V2
pages, predicates + bitmaps, multiple row groups).
### API and Format
No changes under `include/paimon/` and no storage format or protocol change.
The new `PageReader::set_data_page_read_plan` / `DataPageReadPlanEntry` are
experimental APIs added only to the vendored Arrow patch
(`cmake_modules/arrow.diff`), consumed internally by the Paimon Parquet reader.
Read results are unchanged; only the I/O pattern of page-filtered reads is
affected.
### Documentation
No new user-facing feature; behavior and constraints are documented via
comments on the new Arrow API and in `page_filtered_row_group_reader.{h,cpp}`.
### Generative AI tooling
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
