adriangb opened a new issue, #10877: URL: https://github.com/apache/arrow-rs/issues/10877
**Is your feature request related to a problem or challenge?** `VariantMetadata::get` validates the dictionary entry as UTF-8 on every call: https://github.com/apache/arrow-rs/blob/bb0d497b75dbbd2988a1fe527c6e97b5b49b0209/parquet-variant/src/variant/metadata.rs#L369-L372 It does this even when the instance came from `try_new` or `with_full_validation`. Full validation has already proved that the whole dictionary value region is valid UTF-8, and that every adjacent pair of offsets delimits a range that is in bounds, non-decreasing, and lands on character boundaries. `get` sits on the field-access path: `VariantObject::field_name`, `VariantObject::iter`, `VariantMetadata::get_entry` and `impl Index` all route through it, and `get_entry` calls it once per candidate while searching. So a validated metadata buffer pays UTF-8 validation once per field access instead of once per buffer. This shows up in a write-heavy workload that stores semi-structured attributes as Variant, where the same small dictionary of short field names is read back many times. **Describe the solution you'd like** Read the entry directly when `is_fully_validated()` is already true, and keep the checked path for unvalidated instances. **Describe alternatives you've considered** Caching the validated value region as a `&str` inside `VariantMetadata` avoids `unsafe`, but the struct is pinned at 32 bytes on 64-bit by `expect_size_of::<VariantMetadata>(32)` (deliberately, since it is embedded in `VariantList` and `VariantObject`), and a second fat pointer would take it to 48. **Additional context** The invariant this relies on is already property-tested by `validated_metadata_is_accessible` in `parquet-variant/tests/proptest.rs`, added in #10352, which asserts that every entry of validated metadata is readable. # AI usage This issue was written with Claude Code and reviewed by a human. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
