wangyong9999 opened a new pull request, #267: URL: https://github.com/apache/paimon-cpp/pull/267
### Purpose Linked issue: N/A Repeated scans in the same process rebuilt several immutable planning resources: - `DataEvolutionBatchScan` loaded the schema and rebuilt `CoreOptions` after `TableScan` had already initialized them. This also dropped the cache supplied through `ScanContext`. - snapshot live manifest entries were cached only when a bucket filter was present. - each BTree indexer created a new `CacheManager`; closing a reader invalidated its pages, and the next reader opened the index file before checking the cache. **This change reuses the `TableSchema`, `CoreOptions`, filesystem, executor, memory pool, and cache already owned by the scan when planning a DataEvolution global-index query.** **It adds a whole-table snapshot live manifest cache entry, separate from bucket-specific entries. The cache stores live entries before query filtering and applies the current predicate after a cache hit.** **It shares a bounded BTree `CacheManager` between indexers with the same cache configuration, retains immutable pages after reader destruction, and opens the input stream only on a cache miss. Retained pages use the process default memory pool so they do not outlive a caller-owned pool.** The manifest cache remains disabled for row-range scans. Cache read, deserialization, serialization, or write failures fall back to rebuilding the manifest entries. Cache-manager creation is serialized; the existing LRU remains thread-safe and bounded by `btree-index.cache-size` for each distinct cache configuration. The existing eager `BlockCache` constructor and `Close()` behavior are unchanged. On an HDFS primary-key table with 606K rows, 51 active data files, and 6.5 GB of data, a `logid` lookup returning two rows was executed twice in one process. The cold plan/read/total time was 851/163/1015 ms. The second plan/read/total time was 115/99/214 ms. BTree evaluation decreased from 621 ms to 0.55 ms. Both runs selected one indexed split, one data file, and a two-row range, and matched the result of an index-disabled scan. ### Tests - debug build: `core_test`, `common_test`, and `sst_format_test` - `AppendOnlyFileStoreScanTest.TestSnapshotLiveManifestCache*` (4 cases) - `LruCacheTest.TestForSnapshotLiveManifestEntries` - `BlockCacheTest.*` (7 cases) - `LazyFilteredBTreeReaderTest.*` (33 cases) - `*BTree*` (144 passed; 7 existing compatibility-data cases skipped by their fixture guard) - two-query native-HDFS validation: `validation status=PASS`, `path_proof status=PASS path=pk-file-local-btree` - `git diff --check` The local environment does not provide CMake or pre-commit executables, so the CMake and pre-commit checks are left to CI. ### API and Format Adds an overload of `CacheKey::ForSnapshotLiveManifestEntries` for a whole-table cache key. This is an additive API change. There is no object-layout, storage-format, or protocol change. ### Documentation No new user-facing feature or option. ### Generative AI tooling Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
