mkleen opened a new pull request, #11911:
URL: https://github.com/apache/arrow-rs/pull/11911
# Which issue does this PR close?
- Part of #9296.
- This is the `INT32`-only part of #11285, split out for easier review.
# Rationale for this change
Today, getting page-level statistics as Arrow arrays copies the data twice:
- `decode_column_index` reads the Thrift bytes into `ThriftColumnIndex`.
- `PrimitiveColumnIndex::try_new` then copies the mins and maxes into typed
`Vec<T>`s, putting default values in place for null pages.
- `data_page_mins`, `data_page_maxes`, `data_page_null_counts` and
`data_page_nan_counts` go through that `ColumnIndexMetaData` again and push
each value into an Arrow builder one at a time.
- Each call walks the index separately.
# What changes are included in this PR?
This PR adds a way to skip loading the column index, then decode only the
columns you need straight into Arrow arrays. Only `INT32` columns are supported
for now.
### New API
```rust
pub struct DataPageStatistics {
pub mins: ArrayRef,
pub maxes: ArrayRef,
pub null_counts: UInt64Array,
pub nan_counts: UInt64Array,
}
impl StatisticsConverter<'_> {
pub fn data_page_statistics_from_bytes<'b, I>(&self, column_indexes: I)
-> Result<DataPageStatistics>
where I: IntoIterator<Item = (usize, Option<&'b [u8]>)>;
}
```
- **Input:** one `(num_pages, bytes)` pair per row group. The bytes are the
column chunk's serialized `ColumnIndex`, found with
`ColumnChunkMetaData::column_index_range()`. To avoid parsing the index when
the metadata is loaded, load with `PageIndexPolicy::Skip`.
- **Output:** one call returns all four arrays, matching `data_page_mins`,
`data_page_maxes`, `data_page_null_counts` and `data_page_nan_counts`.
- **Missing data:** a row group with `None` bytes, or a column that isn't in
the Parquet file, gives `num_pages` nulls in all four arrays.
- **Unsupported types:** a column whose physical type is not `INT32` returns
a `NYI` error.
The decoder is `parquet/src/arrow/arrow_reader/statistics/page_index.rs`. It
reads the Thrift compact encoding of `ColumnIndex` in one pass, on top of
`ThriftSliceInputProtocol`.
| Field | Thrift type | Handling |
|---|---|---|
| 1 `null_pages` | `list<bool>` | Stored one byte per element (`0x01` =
true; `0x00` and `0x02` = false; any other byte is an error). Inverted to "page
has a min/max" and packed into the `NullBuffer` shared by mins and maxes. |
| 2 `min_values`, 3 `max_values` | `list<binary>` | Each element is decoded
straight into a buffer of the column's physical type (see below). The bytes of
null pages are skipped without being checked. |
| 4 `boundary_order` | `i32` | Read and checked, then discarded. |
| 5 `null_counts`, 8 `nan_counts` | `list<i64>` | Zigzag varints decoded
into `Vec<u64>`. A negative count is an error. A row group without the list
gives null entries. |
| 6, 7 level histograms, and unknown fields | any | Skipped. For integer
lists, skipping just counts the bytes whose top bit is clear, which is much
faster than decoding each varint. |
**Physical type to buffer:**
| Physical type | Buffer | How each value is decoded |
|---|---|---|
| `INT32` | `Vec<i32>` | First 4 bytes, little-endian |
As in the old decoder, a value with extra trailing bytes is accepted, and
one with too few bytes is an error with the same message: `error converting
value, expected 4 bytes got M`.
### Benchmark
Benchmarks can be found here #11287. This PR adds an `Int32` column to them.
These numbers are from a local run.
| Type | Pages | full page index (`main`) | from bytes (branch) | Speedup |
|---|---:|---:|---:|---:|
| Int32 | 2,000 | 30.9 µs | **6.6 µs** | **4.7×** |
| Int32 | 10,000 | 123.8 µs | **28.0 µs** | **4.4×** |
# Are these changes tested?
- **Unit tests** in `page_index.rs`:
- `INT32` is compared against every Arrow type with the existing route.
- Specific tests cover all-null pages, zero pages, no row groups, a column
missing from the file, unknown fields and histograms, mins stored before
`null_pages`, and the `NYI` error for other physical types.
- `parquet/tests/arrow_reader/statistics.rs` now also checks every existing
page-level case for `INT32` columns through `data_page_statistics_from_bytes`,
using the column index bytes read from the file.
# Are there any user-facing changes?
Yes. This PR adds a new public API but nothing existing changes.
- New method `StatisticsConverter::data_page_statistics_from_bytes`.
- New struct `parquet::arrow::arrow_reader::statistics::DataPageStatistics`.
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated content
has been manually reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
