zhf999 opened a new pull request, #243:
URL: https://github.com/apache/paimon-cpp/pull/243
> **Note:** This branch is still under active development. The PR is created
temporarily to
> synchronize progress; it is not ready for review or merge.
This PR is inspired by #242, thanks @gripleaf for the initial work.
### Purpose
This PR introduces a **late-materializing file batch reader**
(`LateMaterializingFileBatchReader`)
that performs probe/payload two-phase reads when a predicate is pushed down
through
`SetReadSchema`.
**Motivation.** In a standard read, all projected columns are materialized
together, even though a
pushed-down predicate may filter out the vast majority of rows. Late
materialization splits the
read into two passes:
1. **Probe pass** — read only the predicate columns, evaluate the predicate
batch by batch, and
build a `matched_bitmap_` of surviving rows.
2. **Payload pass** — read the remaining (payload) columns, compacted to
only the matched rows,
then assemble the probe and payload columns into a single struct array in
`full_schema_` field
order.
This avoids decoding and holding the wide payload columns for rows that will
be discarded.
**Architecture.** The reader is installed **below the prefetch layer** by
wrapping the
format-specific `ReaderBuilder` with a `LateMaterializingReaderBuilder`
(`AbstractSplitRead::CreateFileBatchReader`). Each parallel reader under the
prefetch layer thus
gets its own late-materializing wrapper. The reader implements the full
`PrefetchFileBatchReader` interface, forwarding schema, row-count, seek,
read-range, and metric
queries to the inner reader while interposing the two-phase logic in
`SetReadSchema` /
`NextBatch`.
**Key files:**
- `src/paimon/common/reader/late_materializing_file_batch_reader.{h,cpp}` —
core reader with a
state machine (`kInit → kProbing → kRunning | kNoLatMat → kEOF`), probe
data filtering, payload
batch reading with bitmap compaction, and full-batch assembly.
- `src/paimon/common/reader/late_materializing_reader_builder.h` —
`ReaderBuilder` wrapper that
installs the late-materializing reader around each format reader produced
by the inner builder.
- `src/paimon/core/operation/abstract_split_read.{h,cpp}` — integration: the
reader builder is
wrapped with `LateMaterializingReaderBuilder` inside
`CreateFileBatchReader`.
- `src/paimon/testing/mock/mock_file_batch_reader.h` — enhanced to
faithfully emulate a real
format reader: `SetReadSchema` now resets position and read ranges;
`NextBatchWithBitmap`
honours assigned read ranges; a `ProjectBatch` helper supports column
re-selection.
- `src/paimon/CMakeLists.txt` — registers the new source and test files.
### Tests
A dedicated test suite `LateMaterializingFileBatchReaderTest` is added in
`src/paimon/common/reader/late_materializing_file_batch_reader_test.cpp` (15
cases):
| Test case | Verifies |
|---|---|
| `PassThroughWhenNoPredicate` | No predicate → passthrough, single batch
returned |
| `PassThroughWhenPayloadEmpty` | All fields are probe fields → passthrough |
| `ContiguousSubsetAcrossBatches` | Contiguous matched rows spanning
multiple batches |
| `ScatteredAlternatingMatch` | Non-contiguous alternating matches compacted
correctly |
| `MatchedIntersectsSelection` | Predicate bitmap intersected with
`selection_bitmap` |
| `EmptyMatchReturnsEof` | Zero matched rows → immediate EOF |
| `SeekToRowRealignsProbeCursor` | `SeekToRow` realigns the probe cursor and
state |
| `ReadRangesForwardedAcrossPhases` | Read ranges forwarded to inner reader
across phases |
| `ReentrantSetReadSchema` | Repeated `SetReadSchema` resets state cleanly |
| `ForwardsRowCountAndFileSchema` | `GetNumberOfRows` / `GetFileSchema`
forwarded |
| `MultiFieldPreservesColumnOrder` | Multi-field schema preserves field
order in assembled batch |
| `NestedPayloadColumn` | Nested-type payload column handled correctly |
| `WorksAsInnerOfPrefetchReader` | Works as inner reader of
`PrefetchFileBatchReaderImpl` |
| `PrefetchInnerReentrantSetReadSchema` | Reentrant `SetReadSchema` under
prefetch |
| `PrefetchInnerParallelReadersWithSeek` | Parallel prefetch readers with
seek interleave correctly |
> **Note:** As the branch is still in development, not all validation
commands above have been run
> to completion. The final validation will be performed before marking the
PR ready for review.
### API and Format
No change to public API under `include/paimon/`, no storage format or
protocol change.
The only signature changes are **internal** to the split-read path:
- `AbstractSplitRead::CreateFileBatchReader` — `const ReaderBuilder*` →
`std::unique_ptr<ReaderBuilder>` (ownership transfer to allow wrapping).
- `AbstractSplitRead::CreateFieldMappingReader` — same change for the
`reader_builder` parameter.
Both are private methods of `AbstractSplitRead`; all call sites within the
class are updated in the
same change.
### Documentation
No user-facing documentation change yet. As the feature stabilizes,
documentation for the
late-materialization read mode and its configuration options will be added
in a follow-up.
### Generative AI tooling
Generated-by: Qoder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
