wangyong9999 opened a new issue, #328: URL: https://github.com/apache/paimon-cpp/issues/328
### Problem Selective Parquet reads repeat CPU work within one file reader: Arrow 17 reparses OffsetIndex on each lookup, page planning scans page locations for sparse row selections, and decoding rebuilds a direct-read plan just to determine whether compressed row coordinates apply. The byte cache in #272 reduces repeated I/O but does not eliminate this parsing and planning work. ### Proposed improvement - Reuse parsed OffsetIndex objects within each retained row-group reader. - Seek across page gaps for sparse selections and visit each selected page once. - Reuse the direct-plan decision during decoding. Preserve index validation, dictionary handling, per-leaf row coordinates and missing-index fallbacks. Keep parsed objects reader-local without introducing a shared data cache or public option. ### Validation Check selected values and index lifetimes, including sparse/dense selections and the row-group retention limit. Compare against main with the format benchmark, reporting latency and storage bytes without assuming a general speedup. Implementation: #314. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
