lxy-9602 commented on PR #267: URL: https://github.com/apache/paimon-cpp/pull/267#issuecomment-5491358413
Thanks for working on the repeated Global Index lookup performance. Some downstream query engines, such as StarRocks, have a similar requirement. Their approach is to create a `GlobalIndexReader` through the public API in `global_index_scan.h` and retain/cache the reader for multiple queries. This already preserves the BTree reader's internal block cache without introducing process-wide cache sharing inside paimon-cpp. The process-wide design in this PR introduces additional concepts and hidden global state, including the shared `CacheManager` registry, cache configuration LRU, `cache_namespace`, backend lifetime retention, and cross-reader allocator management. It also makes memory accounting and cache isolation harder for downstream engines that already have their own resource management. Could we instead encourage the query engine to split the query into two explicit stages? 1. **Global Index stage:** create or reuse a `GlobalIndexReader` through `GlobalIndexScan`, evaluate the index predicate, and produce a `GlobalIndexResult`. This stage can be distributed by shard. 2. **Data Scan stage:** pass the returned `GlobalIndexResult` through `ScanContextBuilder::SetGlobalIndexResult()`, then use `TableScan` to generate the data scan plan and execute it. The engine can cache readers using a key containing the table identity, snapshot/index generation, field, index type, shard/partition, row range, and relevant options. It should also ensure that the index lookup and subsequent data scan use the same snapshot, and respect `GlobalIndexReader::IsThreadSafe()` when sharing readers. This keeps cache ownership, eviction, memory accounting, and isolation under the query engine's control while reusing the existing public API. Could we evaluate this approach before adding process-wide cache sharing to paimon-cpp? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
