friendlymatthew opened a new pull request, #10358: URL: https://github.com/apache/arrow-rs/pull/10358
# Rationale for this change Based on benchmark https://github.com/apache/arrow-rs/pull/10357 <img width="1512" height="382" alt="Screenshot 2026-07-16 at 5 15 20 PM" src="https://github.com/user-attachments/assets/74425745-85ea-4411-88f7-6845f6dd8ad7"; /> This PR speeds up field extraction from unshredded variant objects by avoiding repeated field name decoding for every row Variant object keys are stored as numeric field ids that map to a metadata dictionary. `variant_get` currently stores a named path by repeatedly mapping those ids back to strings and comparing the strings for each row, _even when a batch shares one metadata dictionary_. This repeated per-row work dominates when dealing with wide dictionaries This optimization adds metadata scoped field id lookups, caches parsed metadata, and resolves named path elements to ids once per metadata dictionary before scanning rows. It re-resolves paths when the metadata changes and retains name lookup for unsorted dictionaries -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
