cetra3 opened a new pull request, #12309:
URL: https://github.com/apache/arrow-rs/pull/12309

   # Which issue does this PR close?
   
   - Part of #11511
   
   This PR is stacked on #12308. Its own change is the second commit, 
"feat(variant): add header-only views of encoded variant bytes". It is a draft 
until #12308 merges.
   
   # Rationale for this change
   
   Every access to a variant value pays for checks that the access does not 
use. The unvalidated constructors of `VariantMetadata`, `VariantObject` and 
`VariantList` read the first and last offsets of each header and trim the byte 
slices. A field lookup then validates the UTF-8 of every dictionary name that 
its binary search probes.
   
   For most kernel work, a value needs much less: the header of each object or 
list along a path, the names that the binary search probes, and the bytes of 
the value that it finds.
   
   This PR adds the types that read that way, and builds the existing types on 
them. The kernels start to use the views in the next PRs of the stack.
   
   # What changes are included in this PR?
   
   - New types `VariantView`, `VariantMetadataView`, `VariantObjectView` and 
`VariantListView`.
     - Building a view reads one header and nothing else.
     - Each accessor reads and bounds-checks only the bytes that it needs, and 
returns an error on malformed input. It never panics.
     - Field names are compared as bytes, which orders them as `str` does, so a 
search does not validate UTF-8.
     - A view promotes to the existing types: 
`VariantMetadataView::into_metadata`, `VariantObjectView::into_object`, 
`VariantListView::into_list` and `VariantView::to_variant` add the first and 
last offset checks that unvalidated instances guarantee.
   - `VariantMetadata`, `VariantObject` and `VariantList` are now built by 
promoting a view, and their accessors (`get`, `field`, `field_name`, `iter`, 
`get_entry`) delegate to it. Their public behavior is unchanged.
   - `ObjectBuilder::finish` sorts fields with a new 
`MetadataBuilder::cmp_field_names`. `ReadOnlyMetadataBuilder` compares the 
names as bytes.
   
   The docs of `VariantView` describe the design.
   
   # Are these changes tested?
   
   The existing types now delegate to the views, so their tests also cover the 
views. The views also have direct tests: agreement with `VariantMetadata`, 
`VariantObject` and `VariantList`, and every truncation and single-byte 
corruption of a sample value, which must not panic.
   
   Benchmarks (local run, this PR against #12308, x86_64 Linux):
   
   | benchmark | #12308 | this PR | change |
   |---|---:|---:|---:|
   | `bench_validate_complex_object` | 85.43 µs | 90.60 µs | +6% |
   | `bench_validate_large_nested_list` | 8.37 ms | 8.31 ms | −1% |
   | `bench_validate_large_object` | 22.63 ms | 22.85 ms | +1% |
   | `metadata_iter/128` | 561.21 ns | 643.75 ns | +15% |
   | `metadata_iter/32` | 144.04 ns | 164.88 ns | +14% |
   | `metadata_iter/8` | 35.09 ns | 40.37 ns | +15% |
   | `object_field_name/128` | 932.97 ns | 895.70 ns | −4% |
   | `object_field_name/32` | 239.72 ns | 229.85 ns | −4% |
   | `object_field_name/8` | 64.89 ns | 62.90 ns | −3% |
   | `object_get_by_name/128` | 70.30 ns | 56.68 ns | −19% |
   | `object_get_by_name/32` | 44.34 ns | 38.97 ns | −12% |
   | `object_get_by_name/8` | 35.86 ns | 32.99 ns | −8% |
   | `object_iter/128` | 1.83 µs | 1.66 µs | −9% |
   | `object_iter/32` | 467.03 ns | 417.15 ns | −11% |
   | `object_iter/8` | 121.88 ns | 110.17 ns | −10% |
   | `variant_get_unshredded_object_path_262k_rows` | 21.16 ms | 22.79 ms | +8% 
|
   | `variant_get_list_index_unshredded` | 544.94 µs | 576.96 µs | +6% |
   | `variant_get_utf8_from_list` | 342.37 µs | 369.64 µs | +8% |
   | `variant_get_utf8_from_map_in_list` | 349.35 µs | 367.95 µs | +5% |
   | `variant_get_utf8_from_int` | 256.80 µs | 256.21 µs | 0% |
   | `shred_variant_partial_object_8k_rows` | 11.34 ms | 11.11 ms | −2% |
   | `shred_variant_unmatched_object_8k_rows` | 15.15 ms | 14.81 ms | −2% |
   | `unshred_variant_object` | 2.97 ms | 2.93 ms | −1% |
   | `unshred_variant_partial_object` | 24.00 ms | 24.15 ms | +1% |
   | `unshred_variant_partial_object_unsorted_metadata` | 132.94 ms | 107.55 ms 
| −19% |
   
   Slower inputs: `metadata_iter` is 13–16% slower, and it stays slower through 
the rest of the stack. The unshredded `variant_get` benchmarks that walk a path 
(`variant_get_unshredded_object_path_262k_rows`, 
`variant_get_list_index_unshredded`, `variant_get_utf8_from_list`, 
`variant_get_utf8_from_map_in_list`) are 5–8% slower in this PR. The next PR of 
the stack moves `variant_get` onto the views and makes them 11–47% faster than 
they are here. `bench_validate_complex_object` was 6% slower in this run, and I 
have not re-run it.
   
   # Are there any user-facing changes?
   
   New public API in `parquet-variant`:
   
   - `VariantView`, `VariantMetadataView`, `VariantObjectView` and 
`VariantListView`, with their methods.
   - `VariantMetadata::view`, `VariantObject::view` and `VariantList::view`.
   - `MetadataBuilder::cmp_field_names`, a provided trait method.
   - `VariantBasicType`, which `VariantView::basic_type` returns.
   
   No existing public API changes. Some out-of-range errors from metadata and 
list accessors have new message text.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to