gripleaf opened a new pull request, #196:
URL: https://github.com/apache/paimon-cpp/pull/196
### Purpose
Add bounded late materialization for raw-file reads to reduce unnecessary
payload-column I/O.
When enabled, the reader:
- Reads predicate columns first and evaluates predicates, deletion
vectors, and index bitmaps.
- Reads payload columns only for matching row ranges.
- Preserves row offsets and reader metrics when combining probe and
payload readers.
- Falls back to the regular read path when the number of matching rows
exceeds the configured threshold.
- Uses the regular read path when late-materialization prerequisites are
not satisfied.
The following options are introduced:
- read.late-materialization.enabled
- read.late-materialization.max-match-rows
This change also adds validation for malformed predicate results, invalid
bitmap positions, missing payload readers, and inconsistent probe/payload batch
sizes.
### Tests
- Added unit tests for:
- Probe and payload column merging.
- Probe-only reads.
- Payload bitmap filtering.
- Row-offset preservation across batches.
- Reader metrics preservation.
- Payload batches with different lifetimes.
- Mismatched probe and payload batch sizes.
- Missing payload readers and invalid arguments.
- Default, configured, and invalid late-materialization options.
- Ran paimon-common-test with the relevant late-materialization and
metrics tests: 10 tests passed.
- Ran CoreOptionsTest.* and RawFileSplitReadTest.*: 44 tests passed.
- Built paimon-common-test and paimon-core-test in the dev container with
GCC 13.3 and -Werror.
- Ran clang-format --dry-run --Werror and git diff --check.
### API and Format
This change adds public option-name constants under include/paimon/defs.h.
It does not change the storage format or protocol and remains backward
compatible because late materialization is disabled by default.
### Documentation
Added user documentation describing the late-materialization options,
activation requirements, bounded fallback behavior, and metrics semantics.
### Generative AI tooling
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
