cetra3 opened a new pull request, #12309:
URL: https://github.com/apache/arrow-rs/pull/12309
# Which issue does this PR close?
- Part of #11511
This PR is stacked on #12308. Its own change is the second commit,
"feat(variant): add header-only views of encoded variant bytes". It is a draft
until #12308 merges.
# Rationale for this change
Every access to a variant value pays for checks that the access does not
use. The unvalidated constructors of `VariantMetadata`, `VariantObject` and
`VariantList` read the first and last offsets of each header and trim the byte
slices. A field lookup then validates the UTF-8 of every dictionary name that
its binary search probes.
For most kernel work, a value needs much less: the header of each object or
list along a path, the names that the binary search probes, and the bytes of
the value that it finds.
This PR adds the types that read that way, and builds the existing types on
them. The kernels start to use the views in the next PRs of the stack.
# What changes are included in this PR?
- New types `VariantView`, `VariantMetadataView`, `VariantObjectView` and
`VariantListView`.
- Building a view reads one header and nothing else.
- Each accessor reads and bounds-checks only the bytes that it needs, and
returns an error on malformed input. It never panics.
- Field names are compared as bytes, which orders them as `str` does, so a
search does not validate UTF-8.
- A view promotes to the existing types:
`VariantMetadataView::into_metadata`, `VariantObjectView::into_object`,
`VariantListView::into_list` and `VariantView::to_variant` add the first and
last offset checks that unvalidated instances guarantee.
- `VariantMetadata`, `VariantObject` and `VariantList` are now built by
promoting a view, and their accessors (`get`, `field`, `field_name`, `iter`,
`get_entry`) delegate to it. Their public behavior is unchanged.
- `ObjectBuilder::finish` sorts fields with a new
`MetadataBuilder::cmp_field_names`. `ReadOnlyMetadataBuilder` compares the
names as bytes.
The docs of `VariantView` describe the design.
# Are these changes tested?
The existing types now delegate to the views, so their tests also cover the
views. The views also have direct tests: agreement with `VariantMetadata`,
`VariantObject` and `VariantList`, and every truncation and single-byte
corruption of a sample value, which must not panic.
Benchmarks (local run, this PR against #12308, x86_64 Linux):
| benchmark | #12308 | this PR | change |
|---|---:|---:|---:|
| `bench_validate_complex_object` | 85.43 µs | 90.60 µs | +6% |
| `bench_validate_large_nested_list` | 8.37 ms | 8.31 ms | −1% |
| `bench_validate_large_object` | 22.63 ms | 22.85 ms | +1% |
| `metadata_iter/128` | 561.21 ns | 643.75 ns | +15% |
| `metadata_iter/32` | 144.04 ns | 164.88 ns | +14% |
| `metadata_iter/8` | 35.09 ns | 40.37 ns | +15% |
| `object_field_name/128` | 932.97 ns | 895.70 ns | −4% |
| `object_field_name/32` | 239.72 ns | 229.85 ns | −4% |
| `object_field_name/8` | 64.89 ns | 62.90 ns | −3% |
| `object_get_by_name/128` | 70.30 ns | 56.68 ns | −19% |
| `object_get_by_name/32` | 44.34 ns | 38.97 ns | −12% |
| `object_get_by_name/8` | 35.86 ns | 32.99 ns | −8% |
| `object_iter/128` | 1.83 µs | 1.66 µs | −9% |
| `object_iter/32` | 467.03 ns | 417.15 ns | −11% |
| `object_iter/8` | 121.88 ns | 110.17 ns | −10% |
| `variant_get_unshredded_object_path_262k_rows` | 21.16 ms | 22.79 ms | +8%
|
| `variant_get_list_index_unshredded` | 544.94 µs | 576.96 µs | +6% |
| `variant_get_utf8_from_list` | 342.37 µs | 369.64 µs | +8% |
| `variant_get_utf8_from_map_in_list` | 349.35 µs | 367.95 µs | +5% |
| `variant_get_utf8_from_int` | 256.80 µs | 256.21 µs | 0% |
| `shred_variant_partial_object_8k_rows` | 11.34 ms | 11.11 ms | −2% |
| `shred_variant_unmatched_object_8k_rows` | 15.15 ms | 14.81 ms | −2% |
| `unshred_variant_object` | 2.97 ms | 2.93 ms | −1% |
| `unshred_variant_partial_object` | 24.00 ms | 24.15 ms | +1% |
| `unshred_variant_partial_object_unsorted_metadata` | 132.94 ms | 107.55 ms
| −19% |
Slower inputs: `metadata_iter` is 13–16% slower, and it stays slower through
the rest of the stack. The unshredded `variant_get` benchmarks that walk a path
(`variant_get_unshredded_object_path_262k_rows`,
`variant_get_list_index_unshredded`, `variant_get_utf8_from_list`,
`variant_get_utf8_from_map_in_list`) are 5–8% slower in this PR. The next PR of
the stack moves `variant_get` onto the views and makes them 11–47% faster than
they are here. `bench_validate_complex_object` was 6% slower in this run, and I
have not re-run it.
# Are there any user-facing changes?
New public API in `parquet-variant`:
- `VariantView`, `VariantMetadataView`, `VariantObjectView` and
`VariantListView`, with their methods.
- `VariantMetadata::view`, `VariantObject::view` and `VariantList::view`.
- `MetadataBuilder::cmp_field_names`, a provided trait method.
- `VariantBasicType`, which `VariantView::basic_type` returns.
No existing public API changes. Some out-of-range errors from metadata and
list accessors have new message text.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]