gripleaf opened a new pull request, #310:
URL: https://github.com/apache/paimon-cpp/pull/310
### Purpose
Linked issue: #240
Reduce unnecessary decoding and Arrow materialization when reading a
single bucket from a mixed-bucket manifest.
- When manifest bytes are retained in memory, first read the version and
bucket fields, validate every entry’s version, and collect matching row IDs.
Then materialize only matching entries using precise bitmap selection.
- Support precise bitmap selection in the Avro reader while preserving
original file row IDs.
- Reuse reader-local block indexes to skip blocks without matching rows.
Limit indexes to 65,536 blocks and fall back to sequential selection when the
limit is exceeded or the initial pass is incomplete.
- Preserve single-pass reads when bytes are not retained, precise
selection is unavailable, or manifest metadata identifies a single matching
bucket.
This change does not include the separate Avro skip-plan optimization. No
isolated end-to-end latency improvement is claimed.
### Tests
Added coverage for:
- Empty, sparse, full, and out-of-range bitmap selections.
- Nested nullable values, original row IDs, schema resets, and repeated
EOF.
- Cross-block selection after complete and incomplete projection passes.
- Sequential fallback after exceeding the block-index limit.
- Cached and uncached bucket reads.
- Invalid versions and null version/bucket fields outside the selected
bucket.
Local validation:
- Avro tests: 73/73 passed.
- Manifest and scan tests: 43/43 passed.
- Read integration tests: 294/294 passed.
- Core tests: 1,899/1,901 passed. Two commit-path tests failed because
their fixtures select ORC manifests, which the writer rejects as read-only. A
separate clean-main reproduction was not run.
- Changed-file pre-commit checks and git diff --check passed.
For local GCC 13 validation, -Wmaybe-uninitialized was downgraded from an
error only for the unchanged arrow_realtime_store_test.cpp generated build
rule. No warning-policy changes are included.
### API and Format
No public API signatures, storage formats, or protocols are changed.
The Avro reader now honors the existing selection bitmap API and reports
precise bitmap-selection support. Block indexes are transient, bounded,
reader-local state.
### Documentation
No new user-facing configuration is introduced. Implementation comments
describe the retained-bytes requirement, bounded indexes, and fallback behavior.
### Generative AI tooling
Generated-by: OpenAI Codex (GPT-5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
