cetra3 opened a new pull request, #12311: URL: https://github.com/apache/arrow-rs/pull/12311
# Which issue does this PR close? - Part of #11511 This PR is stacked on #12308, #12309 and #12310. Its own change is the fourth commit, "perf(variant): copy leftover object fields by field id when shredding". It is a draft until they merge. # Rationale for this change When `shred_variant` and `unshred_variant` copy an object field that is not shredded, they decode it into a `Variant`, look its name up in the metadata dictionary to get the field id back, and encode the value again. The output uses the same metadata dictionary as the input, so the field id and the encoded bytes are already correct. # What changes are included in this PR? - `ObjectBuilder::insert_encoded` adds a field by its field id and its encoded bytes. - `VariantView::bytes` returns the exact encoding of a value. - `shred_variant` and `unshred_variant` copy leftover object fields through them. They match field names as bytes against a sorted list of the shredded field names. # Are these changes tested? The existing shredding tests cover the changed code. `VariantView::bytes` has tests for every primitive type, and for the fields of an object and the elements of a list, whose encodings must tile the value region of their parent. Benchmarks (local run, this PR against #12310, x86_64 Linux): | benchmark | #12310 | this PR | change | |---|---:|---:|---:| | `shred_variant_partial_object_8k_rows` | 10.90 ms | 9.42 ms | −14% | | `shred_variant_unmatched_object_8k_rows` | 14.43 ms | 11.36 ms | −21% | | `unshred_variant_list` | 5.44 ms | 5.59 ms | +3% | | `unshred_variant_list_of_objects` | 11.26 ms | 11.46 ms | +2% | | `unshred_variant_nested_fallback` | 74.62 ms | 75.06 ms | +1% | | `unshred_variant_object` | 2.93 ms | 3.05 ms | +4% | | `unshred_variant_partial_object` | 24.13 ms | 21.81 ms | −10% | | `unshred_variant_partial_object_unsorted_metadata` | 107.97 ms | 62.79 ms | −42% | | `unshred_variant_primitive` | 263.33 µs | 260.74 µs | −1% | | `variant_get_unshredded_object_path_262k_rows` | 14.98 ms | 13.61 ms | −9% | | `variant_get_list_index_unshredded` | 308.11 µs | 283.84 µs | −8% | | `variant_get_utf8_from_unshredded_string` | 214.52 µs | 233.12 µs | +9% | | `variant_get_binary_from_string` | 207.79 µs | 226.29 µs | +9% | Slower inputs: `unshred_variant_object` is 4% slower and `unshred_variant_list` is 3% slower. `variant_get_utf8_from_unshredded_string` and `variant_get_binary_from_string` are 9% slower in this run, although this PR does not change `variant_get`; these two benchmarks have moved by 10–15% with unrelated changes on this machine. # Are there any user-facing changes? New public API in `parquet-variant`: `ObjectBuilder::insert_encoded` and `VariantView::bytes`. On malformed variant bytes, `shred_variant` and the leftover copy in `unshred_variant` return an error. Before, these paths could panic. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
