cetra3 opened a new pull request, #11511:
URL: https://github.com/apache/arrow-rs/pull/11511
# Which issue does this PR close?
- N/A
This PR supersedes #11505 and #11506. It makes the same changes in a general
form.
# Rationale for this change
Every access to a variant value pays for checks that the access does not
use. `Variant::new` and the unvalidated constructors of `VariantMetadata`,
`VariantObject` and `VariantList` read the first and last offsets of each
header and trim the byte slices. A field lookup then validates the UTF-8 of
every dictionary name that its binary search probes. Kernels such as
`variant_get` construct a full `Variant` for every row, and then walk the path
through it.
For most kernel work, a value needs much less: the header of each object or
list along a path, the names that the binary search probes, and the bytes of
the value that it finds.
# What changes are included in this PR?
- A new module, `parquet_variant::view` (`#[doc(hidden)]`), adds
`MetadataView`, `ValueView`, `ObjectView` and `ListView`.
- Building a view reads one header and nothing else.
- Each accessor reads and bounds-checks only the bytes that it needs, and
returns an error on malformed input. It never panics.
- Field names are compared as bytes, which orders them as `str` does, so a
search does not validate UTF-8.
- A view promotes to the existing types: `MetadataView::into_metadata`,
`ObjectView::into_object`, `ListView::into_list` and `ValueView::to_variant`
add the first and last offset checks that unvalidated instances guarantee.
- `VariantMetadata`, `VariantObject` and `VariantList` are now built by
promoting a view, and their accessors (`get`, `field`, `field_name`, `iter`,
`get_entry`) delegate to it. Their public behavior is unchanged.
- `variant_get` reads unshredded columns through the views, for field and
index paths and for typed and `Variant` output. A row reads only the headers
and names along the path, and decodes only the value that it finds.
- `shred_variant` and `unshred_variant` copy leftover object fields by field
id with their encoded bytes, through a new `#[doc(hidden)]`
`ObjectBuilder::insert_encoded`. A leftover field is no longer decoded into a
`Variant` and then re-encoded.
- `ObjectBuilder::finish` sorts fields with a new
`MetadataBuilder::cmp_field_names`. `ReadOnlyMetadataBuilder` compares the
names as bytes.
- `VariantMetadata::borrowed_field_id` is removed. No caller in the
repository needs it after the changes above. Its tests now cover `get_entry`.
# Are these changes tested?
The existing tests of `parquet-variant`, `parquet-variant-compute`,
`parquet-variant-json`, and the `parquet` variant integration tests pass. The
public types now delegate to the views, so their tests also cover the views.
Benchmarks (local run, `main` vs this PR, back to back, Apple Silicon
laptop):
| benchmark | main | this PR | change |
|---|---:|---:|---:|
| `variant_get_unshredded_object_path_262k_rows` | 21.93 ms | 11.98 ms |
−45% |
| `variant_get_list_index_unshredded` | 573.4 µs | 266.5 µs | −54% |
| `variant_get_utf8_from_int` | 272.4 µs | 172.8 µs | −37% |
| `variant_get_utf8_from_unshredded_string` | 336.0 µs | 226.5 µs | −33% |
| `variant_get_binary_from_unshredded_binary` | 283.0 µs | 179.2 µs | −37% |
| `variant_get_utf8_from_list` | 364.6 µs | 329.3 µs | −10% |
| `variant_get_utf8_from_map_in_list` | 376.4 µs | 331.1 µs | −12% |
| `shred_variant_partial_object_8k_rows` | 14.03 ms | 9.29 ms | −34% |
| `shred_variant_unmatched_object_8k_rows` | 18.50 ms | 11.65 ms | −37% |
| `variant_get_list_index_shredded` | 59.4 µs | 58.8 µs | −1% |
| `variant_get_shredded_utf8` | 343.1 ns | 343.6 ns | +0% |
| `object_get_by_name/128` | 86.0 ns | 69.1 ns | −20% |
| `object_iter/128` | 2.0 µs | 1.6 µs | −21% |
| `object_field_name/128` | 1.1 µs | 962.6 ns | −12% |
| `metadata_iter/128` | 688.5 ns | 609.8 ns | −11% |
| `bench_validate_large_object` | 19.39 ms | 19.46 ms | +0% |
| `bench_validate_large_nested_list` | 7.18 ms | 7.34 ms | +2% |
The 16 `variant_get_decimal_*` benchmarks are 25–36% faster. No benchmark in
`variant_kernels` or `variant_validation` is more than 2% slower.
On real-world telemetry data with per-row metadata, `variant_get` on
unshredded object fields is 1.6–1.9x faster than `main` for keys that most rows
contain, and 2.2–2.4x faster for keys that are rare or absent.
# Are there any significant user-facing changes?
No public API changes, other than the new `#[doc(hidden)]` items. Behavior
differences:
- On malformed variant bytes, `variant_get`, `shred_variant` and the
residual copy in `unshred_variant` return an error, or a null under safe
casting. Before, these paths could panic.
- Some out-of-range errors from metadata and list accessors have new message
text.
- A caller outside the repository that passes field names borrowed from the
metadata dictionary to `ObjectBuilder::insert_bytes` now resolves them through
`get_entry` and the existing name cache, not through `borrowed_field_id`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
