leaves12138 opened a new pull request, #9695: URL: https://github.com/apache/paimon/pull/9695
### Purpose [WIP] Reduce remote I/O overhead for Parquet point lookups, particularly when a global BTree index already provides row IDs. - Add a default length-aware `FileIO.newInputStream(Path, long)` overload and forward it through the relevant FileIO proxies. Implementations without the optimization retain the existing path. - Use lazy, exact OSS range reads when file length is known, avoiding an initial HEAD/read-ahead and supporting bounded vectored reads. Preserve Hadoop read statistics and error/close semantics. - Reuse the current data-evolution schema instead of fetching it again. - Apply row-ID row-group selection before dictionary/Bloom predicate filtering, retaining the conservative fallback for missing row offsets. - Buffer at most 128 KiB of the tail per opened remote Parquet file, allowing footer and nearby page indexes to share a range read. Preserve native vectored reads outside that buffer and caller-provided range buffers. This is a read-path change; it does not change the data/index file formats or disable predicate checks. The draft status is intentional for API and connector review. Earlier synthetic tests on two fixed 100-billion-row snapshots, using identical UUIDs and Java 11 clients, showed: | Same-table comparison | Before P50 | Optimized P50 | |---|---:|---:| | Full point lookup, table with page indexes | 953.00 ms | 177.63 ms | | Full point lookup, table without page indexes | 3674.48 ms | 955.00 ms | For one representative page-index lookup, Parquet I/O decreased from 26 GETs + 1 HEAD to 7 GETs, and response-body bytes decreased from 11.71 MB to 4.25 MB. These are client API measurements, not SQL-service end-to-end latency or a production SLA. They were collected on the earlier benchmarked base; this draft is rebased onto current master. ### Tests - 111 targeted tests passed on Linux for Parquet, data-evolution reads, OSS range streams, vectored reads, REST FileIO and caching FileIO. - Review regressions reproduced against the pre-fix implementation: cached vectored reads must fill and return the supplied buffer (including zero-length ranges); closing one failing request must not prevent other requests from being closed, including repeated identical exceptions and unchecked failures. - Current/historical schemas, missing row offsets, predicate filtering, EOF/truncation, close during read, random seeks, tail boundaries and Hadoop read statistics. - Earlier real-table validation: 4,048 checked point lookups, plus before/after whole-file reads of two 16,934,000-row Parquet files with identical ordered content SHA-256 digests. - After the review fixes and rebase, four real-snapshot point-lookup smoke checks passed using the freshly packaged bundle and OSS connector. - Spotless, Checkstyle and `git diff --check`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
