leaves12138 opened a new pull request, #9919: URL: https://github.com/apache/paimon/pull/9919
### Purpose Unify the whole-manifest entry cache and the compressed Avro block cache used by sidecar-selected reads into one decoded-entry cache at physical block granularity. This draft is based on current master, after #9897 and #9908. It contains only the Java cache changes and their tests/documentation, not the deferred Python writer integration from #9743. ### Changes - Change `ManifestEntryCache` to `ObjectsCache<BlockKey, ManifestEntry, ManifestEntrySegments>`, keyed by manifest path, offset and encoded block length. - Let full reads and sidecar-selected reads reuse complete, unfiltered decoded blocks. Apply each query's predicates and converter after retrieving the block. - Remove the separate compressed-block cache from `SelectedBlockInput`; keep the independent sidecar metadata-byte cache unchanged. - Cache a complete physical block directory after reaching EOF or selecting all blocks, so a subsequent full read can find previously cached entries. - Preserve physical entry order and ADD/DELETE semantics. Account for decoded buffers and lookup metadata, and replay an oversized block without retaining a partial cache entry or re-reading the file. - Add coverage for full/selected cache reuse, distinct keys, filters, ordering, failure paths, memory limits and concurrent queries. ### Scope No manifest/sidecar on-disk format or writer behavior changes. The `cache.manifest-sidecar.max-memory` option and its default are unchanged. This is a draft for review of the cache design and implementation; production performance validation is not claimed here. ### Validation - Formatting and `git diff --check` passed. - Targeted Maven reactor tests passed on the rebased branch: core 369 tests (1 skipped), common 2 tests, with no failures or errors. - Command: `mvn -pl paimon-core -am -DwildcardSuites=none -Dtest=Manifest*Test,ObjectsCacheTest,SegmentsCacheTest,DataPagedOutputSerializerTest,DataEvolutionBatchScanTest,CoreOptionsTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false test`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
