gripleaf opened a new pull request, #450: URL: https://github.com/apache/paimon-cpp/pull/450
### Purpose Related: #430 (background). Manifest file cache hits currently read Arrow IPC batches and inspect partition fields row by row. Snapshot live-entry caches separately serialize merged entries. This change introduces a shared `IndexedManifestEntries` representation so both paths can skip unmatched partition/bucket groups before constructing `ManifestEntry` and `DataFileMeta` objects. - Cache complete data manifest files by their existing whole-file keys, allowing reuse across snapshots and query filters. Preserve original ADD/DELETE ordering with row ordinals. - Reuse the indexed representation in `SnapshotLiveManifestEntries`, retaining its bucket-level keys, snapshot retention, and merged live-entry semantics. - Keep historical or unknown bucket layouts eligible for inferred-bucket filtering. Rebuild obsolete or malformed cache envelopes, propagate predicate errors without retry, and retain allocator ownership with accurate cache weight accounting. Cold cache loads still read the complete manifest. Manifest-list and index-manifest caches continue to use Arrow IPC. This draft does not claim a measured performance improvement; benchmarks remain to be run. ### Tests Added coverage for partition/bucket pruning, interleaved ADD/DELETE ordering, inferred bucket layouts, snapshot reuse, malformed envelopes, complete-file reuse across readers and filters, and predicate error propagation. Updated existing cache lifetime, admission, concurrency, and memory accounting tests. Validation completed locally: - `cmake --build build --target paimon-core-test -j 16` — passed. - `./build/debug/paimon-core-test` — 2,364 tests passed before the final row-range overflow-expression correction. - `./build/debug/paimon-core-test --gtest_filter='*Manifest*:*FileStoreScan*:*InferredBucket*'` — 131 tests passed after that final correction and rebuild. - `pre-commit run --files` for all 17 changed files — passed. - `git diff --check` — passed. The standalone integration suite and performance benchmarks have not been run. ### API and Format No changes to public headers under `include/paimon`, configuration options, persisted manifest formats, or protocols. Existing cache kinds and keys are retained. Private data-manifest and snapshot-live cache encodings change; incompatible cached values are rebuilt from the source. File and snapshot caches remain separate logical cache layers using a shared representation. ### Documentation Updated `docs/source/user_guide/manifest_cache.rst` and `docs/source/user_guide/manifest_entry_cache.rst` to describe the shared indexed representation, filtering, cache fallback, and memory accounting. ### Generative AI tooling Generated-by: OpenAI Codex CLI 0.163.0 (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
