XiaoHongbo-Hope opened a new pull request, #8893:
URL: https://github.com/apache/paimon/pull/8893
### Purpose
Data-evolution scans can already carry exact global row-id ranges in an
`IndexedSplit`. Native BLOB, Vortex, Lance, and ROW readers consume those
ranges directly, but the PyArrow Parquet path reads every row in each
intersecting file and applies `RowIdFilterRecordBatchReader` afterwards.
That fallback causes severe read amplification after compaction creates a
large Parquet file: a sparse indexed lookup still decodes the whole file and
executes a Python range check for every row.
This change adds native Parquet row-range reads at row-group granularity:
- translate global row-id ranges to file-local row indices;
- open only the Parquet row groups intersecting those indices;
- slice emitted batches to the exact requested rows while preserving order;
- pass the selected indices to `DataFileBatchReader` so generated `_ROW_ID`
values stay aligned;
- skip `RowIdFilterRecordBatchReader` on this native path.
Correctness guard: the optimization is enabled only when there is no
scanner-level residual predicate. A scanner predicate removes rows before
position-based slicing and could shift positions, so that case deliberately
keeps the existing whole-file read plus row-id filter. Full scans and other
formats are unchanged. Projected VARIANT columns use the same selected row
groups and exact slicer.
### Read amplification reproduction
Read-only reproduction against a compacted data-evolution BLOB table, using
an
indexed equality filter that selected 713 descriptor rows from one Parquet
file containing 24,836,412 rows:
| Metric | Before | After |
| --- | ---: | ---: |
| Returned rows | 713 | 713 |
| Descriptor bytes | 181,815 | 181,815 |
| Per-row Python range checks | 24,836,412 | 0 |
| Descriptor read time (warm environment) | 13.390 s | 0.876 s |
The ordered result SHA-256 was identical before and after.
### Tests
```text
python -m unittest \
pypaimon.tests.parquet_row_range_test \
pypaimon.tests.format_pyarrow_variant_row_group_test \
pypaimon.tests.projection_predicate_index_test \
pypaimon.tests.data_evolution_test
Ran 59 tests in 4.797s
OK
```
New tests cover intersecting-row-group selection, disjoint ranges, batch and
row-group boundaries, payload alignment, bypassing the per-row Python filter,
residual column-predicate fallback, projected VARIANT reads, and unchanged
full scans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
