cetra3 opened a new pull request, #11505: URL: https://github.com/apache/arrow-rs/pull/11505
# Which issue does this PR close? - N/A # Rationale for this change `VariantObject::get` and `VariantMetadata::get_entry` look up a field by name. Each probe of the binary search (or linear scan) reads a dictionary entry through an accessor that returns `&str`. When the metadata has not been fully validated, those accessors UTF-8 validate every name they return. That is the common case for `variant_get` over arrays with per-row metadata, such as the output of `json_to_variant` or data read from Parquet. A lookup only compares names, so this validation is unnecessary. Byte order is `str` order. An entry whose bytes equal the requested name is valid UTF-8, because the requested name is a `&str`. In a profile of `variant_get_unshredded_object_path_262k_rows`, UTF-8 validation from these probes (`core::str::from_utf8` and `VariantMetadata::get`) is about 5–7% of samples. On real-world telemetry data with per-row metadata, we measured `variant_get` on unshredded object fields at 10–25% faster with this change. Lookups for keys that are rare or absent gain the most, because they probe more names. # What changes are included in this PR? - `VariantMetadata::get_bytes` (crate-private) returns the raw bytes of a dictionary entry. Offsets are still bounds-checked. - `VariantObject::get` binary-searches by comparing those bytes with the requested name. - `VariantMetadata::get_entry` compares bytes in both its binary search and its linear scan, and converts only the matching entry to `&str`. # Are these changes tested? Existing tests of `VariantObject::get`, `VariantMetadata::get_entry`, `variant_get` and shredding cover these code paths. Benchmarks (`variant_kernels`, local run, main vs this PR): | benchmark | main | this PR | change | |---|---:|---:|---:| | `variant_get_unshredded_object_path_262k_rows` | 22.28 ms | 21.43 ms | −3.8% | | `shred_variant_unmatched_object_8k_rows` | 18.88 ms | 18.12 ms | −4.0% | | `shred_variant_partial_object_8k_rows` | 14.08 ms | 13.90 ms | −1.3% | Local run on an Apple Silicon laptop. The remaining `variant_kernels` benchmarks don't call these functions and moved within run-to-run noise. # Are there any significant user-facing changes? No. A dictionary name that is not valid UTF-8 now fails to match a lookup, where before it panicked mid-search. Malformed offsets still panic, as before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
