wangyong9999 opened a new issue, #290: URL: https://github.com/apache/paimon-cpp/issues/290
### Search before asking - [x] I searched existing issues. #225 concerns page row-count limits and #214 concerns index pushdown; neither covers this metadata-I/O reuse improvement. ### Motivation Repeated full-primary-key lookups against Parquet files on remote storage still perform avoidable metadata I/O: - Readers reuse cached footers but reload ColumnIndex/OffsetIndex bytes across reader instances. - Arrow's default ColumnIndex read envelope can include large payload-column min/max values even when only primary-key columns participate in the predicate. - Latest-snapshot discovery probes the snapshot directory before reading `LATEST`, although the successful hint path does not require this probe and the listing fallback already checks directory existence. These are performance opportunities, not a claim that the current lookup returns incorrect rows. ### Solution Reuse immutable page-index bytes through the existing caller-provided cache, retain the allocation's memory pool for the cached bytes' lifetime, and limit predicate index reads to predicate columns. Keep query-local readers and clear restricted read hints before loading projected payload offset indexes. Remove the redundant snapshot-directory probe while preserving the next-snapshot check and listing fallback. No process-wide resource manager, snapshot TTL, result cache, public API change or file-format change is proposed. Implementation: #285. ### Anything else? A synthetic remote-HDFS point-lookup experiment measured the incremental snapshot-probe change on an unchanged small-page table. At eight concurrent KV clients, mean latency decreased from 51.24 to 47.79 ms (6.7%), P99 from 90.18 to 81.31 ms, and throughput increased from 155.93 to 167.22 queries/s (7.2%). Both revisions already included page-index reuse and predicate-only index reads; these numbers are not the total PR speedup. The PR records the configuration and measurement limits. A focused regression fixture also reduces predicate-index bytes read from 6,651 to 2,212 while preserving the projected payload result. This is an index-read byte count, not total query I/O. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! See #285. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
