gripleaf opened a new pull request, #212:
URL: https://github.com/apache/paimon-cpp/pull/212
### Purpose
Linked issue: close #xxx
Optimize snapshot live manifest cache rebuilding for bucket-filtered scans.
Previously, rebuilding a bucket-scoped cache entry deserialized every
manifest entry before filtering by bucket. This change reads the serialization
version and bucket first, and performs full `ManifestEntry` deserialization
only for entries belonging to the target bucket.
The optimization is enabled by default and can be disabled with
`scan.manifest-entry.lazy-decode.enabled`.
This change also:
- Preserves serialization-version validation before accessing
version-dependent fields.
- Reuses cached raw manifest bytes when reading bucket entries.
- Ensures manifest readers are closed when batch processing exits early.
- Retains the original full-deserialization path as a configurable
fallback.
### Tests
Added or updated unit tests covering:
- Default and explicitly configured values of
`scan.manifest-entry.lazy-decode.enabled`.
- Manifest serialization-version validation.
- Reading entries for individual buckets.
- Reading a nonexistent bucket and returning an empty result.
- Verifying that entries from non-target buckets are not fully
deserialized.
- Raw manifest byte-cache reuse across bucket reads.
- Snapshot live manifest cache hits and cache-miss rebuilding.
- Result equivalence when lazy decoding is disabled.
- Compatibility with Java Paimon 0.9 and 1.1 Avro manifests.
Validation commands:
```bash
cmake --build build --target paimon-core-test -j 4
./build/debug/paimon-core-test \
--gtest_filter='ManifestFileTest.TestReadBucketEntries*:AppendOnlyFileStoreScanTest.TestSnapshotLiveManifestCache*:ManifestEntrySerializerTest.TestValidateVersion:CoreOptionsTest.TestDefaultValue:CoreOptionsTest.TestFromMap'
clang-format --dry-run --Werror \
include/paimon/defs.h \
src/paimon/core/manifest/manifest_entry_serializer.h \
src/paimon/core/manifest/manifest_file.h \
src/paimon/core/manifest/manifest_file_test.cpp
git diff --check
All 8 selected unit tests passed.
### API and Format
This change adds the following public configuration API:
- Options::SCAN_MANIFEST_ENTRY_LAZY_DECODE_ENABLED
- CoreOptions::ScanManifestEntryLazyDecodeEnabled()
The corresponding option key is
scan.manifest-entry.lazy-decode.enabled, and its default value is true.
This change does not modify the manifest storage format, serialization
version, or external protocol.
### Documentation
The new configuration option and affected internal APIs include English
inline documentation.
No standalone user documentation is added because this option controls an
internal scan optimization and does not change scan results.
### Generative AI tooling
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
