XiaoHongbo-Hope opened a new pull request, #9962: URL: https://github.com/apache/paimon/pull/9962
## What - Add an opt-in Parquet OffsetIndex path for contiguous row-ID selections. PyPaimon leaves it disabled by default and falls back to the ordinary reader for unsupported or unprofitable selections. - Decode only selected encoded pages through PyArrow. Support flat columns and the standard two-binary-field VARIANT representation (`value`, `metadata`), including nullable values. - Reuse `parquet.filter.columnindex.enabled`; this does not add ColumnIndex predicate filtering or change the table format. ## Validation - Related read tests: 108 passed, 18 subtests passed (PyArrow 18); page-index tests: 61 passed (PyArrow 19). Python lint, Python 3.6 syntax check, and `git diff --check` passed. - Read-only remote-file A/B on a representative ~80 MiB Parquet file: one-row result was identical; the ordinary path took ~2.3–2.6 s and transferred ~85 MB, while the opt-in path took ~1.4–1.5 s and transferred <1 MB. GET count increased from 5 to 9–12. This measures the file reader, not an end-to-end application query. This is a draft because the remote I/O behavior and fallback thresholds deserve community review before enabling it by default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
