cetra3 commented on PR #10358: URL: https://github.com/apache/arrow-rs/pull/10358#issuecomment-6086383082
We ran some benchmarks against this PR using arrays where each row has its own metadata dictionary, which is what `json_to_variant` and `VariantArrayBuilder` produce. In that case `variant_get` with an object path is about 2–4x slower than on `main`. The wider the dictionary, the larger the slowdown. From what we can see, there are three reasons: 1. **The cache misses on almost every row.** It's keyed on the metadata pointer, and with per-row metadata every row has a different pointer, even when the bytes are identical. 2. **Every miss fully validates the metadata, and does it twice.** It calls `VariantMetadata::try_new` once in the `variant_get` loop and again in `VariantPathRowBuilder`. `main` doesn't fully validate the dictionary on this path, so the cost of that check across all names lands on every row. 3. **Unsorted dictionaries still pay for path resolution.** Each miss runs a linear `get_entry` scan and allocates a `String`, then falls back to a by-name lookup anyway. The existing `variant_get_unshredded_object_path_262k_rows` benchmark shares one metadata buffer across all rows, so the cache always hits and this case doesn't show up there. A benchmark with per-row metadata, for example `json_to_variant` over objects with varied key sets, would cover it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
