iceberg-kitten opened a new pull request, #1632:
URL: https://github.com/apache/arrow-go/pull/1632
### Rationale for this change
Callers that already know which rows they want can seek to the beginning of
each range. A normal batch can extend past the end of that range, so the caller
still decodes rows it will discard. This comes up when reading payload columns
for a selective query. Related to #38.
### What changes are included in this PR?
- Add `GetRecordReaderWithRowRanges` for sorted, non-overlapping, half-open
ranges relative to the selected row groups.
- Seek across gaps and cap each batch at its range boundary. Projections
with repeated columns read through gaps sequentially to preserve row alignment,
including on files affected by #1522.
- Coalesce adjacent ranges and use the regular reader path for a selection
that covers every row.
### Benchmark
Read a 262,144-row Snappy file with four row groups, an `int64` ID, and a
128-byte fixed-width payload. The selective workloads keep one contiguous range
per row group. All three paths use a 4,096-row batch size and verify the same
selected row count and checksum.
Values below are **ms/op / MiB read from the source / MiB allocated**.
Medians come from 12 fresh processes per method with rotating method order,
pinned to one efficiency core on an Intel i9-12900HK, Linux amd64, Go 1.26.10.
The file is held in memory. Fixture construction is outside the timer.
| Page index | Rows selected | Full decode, then select | Seek, then slice
batches | Row-range reader |
| --- | ---: | ---: | ---: | ---: |
| Yes | 1% | 22.78 / 33.01 / 71.62 | 2.03 / 2.26 / 6.63 | 1.06 / 0.76 / 2.86
|
| Yes | 10% | 22.26 / 33.01 / 71.62 | 3.01 / 4.26 / 10.96 | 2.57 / 3.76 /
8.83 |
| Yes | 100% | 22.70 / 33.01 / 71.62 | 21.60 / 33.01 / 71.62 | 22.22 / 33.01
/ 71.61 |
| No | 1% | 22.45 / 33.07 / 71.72 | 4.70 / 18.79 / 6.76 | 3.63 / 17.29 /
2.99 |
| No | 10% | 22.09 / 33.07 / 71.73 | 5.97 / 20.80 / 11.10 | 5.55 / 20.30 /
8.97 |
| No | 100% | 22.70 / 33.07 / 71.72 | 21.88 / 33.07 / 71.72 | 22.62 / 33.07
/ 71.72 |
For the indexed 1% case, the range reader cut latency by 95% versus full
decode and 48% versus seeking and slicing. At 100% selection it read the same
number of bytes as the two baselines, and its timings were within 3.4% of each
baseline.
To reproduce on Linux, select a local CPU and run the benchmark. For the
measurements above, each method was filtered by its `method=` sub-benchmark and
run in a separate process.
```sh
taskset -c 19 env GOMAXPROCS=1 GOGC=100 go test ./parquet/pqarrow \
-run '^$' -bench '^BenchmarkRecordReaderRowRanges$' \
-benchmem -benchtime=200ms -count=12
```
### Are these changes tested?
`go test -short -tags=assert,test ./parquet/pqarrow ./parquet/file` passed
with the repository test data. The range tests cover empty, full, and invalid
selections, reordered row groups, sequential and parallel reads, seeks after
EOF, and nullable repeated values with and without page indexes.
### Are there any user-facing changes?
The new reader method accepts `[]pqarrow.RowRange`. `SeekToRow` on a range
reader takes an index into the selected rows. Batches can end at a range
boundary before reaching the configured batch size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]