cetra3 opened a new pull request, #11506: URL: https://github.com/apache/arrow-rs/pull/11506
# Which issue does this PR close? - N/A # Rationale for this change For a path of field names over unshredded data, `variant_get` builds a `Variant` for every row, validates its metadata, value and object headers, and then walks the path through it. Most of that work serves fields that the lookup never reads. Rows that do not contain the requested field do all of it and produce a null. In a profile of `variant_get_unshredded_object_path_262k_rows`, per-row `Variant` construction and validation, per-row column downcasts, and the generic path walk take most of the time. The field lookup itself is a small part. # What changes are included in this PR? - A private module, `variant_field_lookup`, finds a field by a path of names directly in the encoded metadata and value bytes. It reads only the headers it needs and the dictionary names that the binary search probes. It compares names as bytes. Every read is bounds-checked. - `variant_get` uses it when the output type is concrete, the path has only field names, the target has no `typed_value`, and the `metadata` and `value` columns are `BinaryView`. It builds a `Variant` only for the value found, and uses the existing row builder to convert it. Rows without the field append null and build nothing. - Other inputs keep the existing per-row path. There are no public API changes. # Are these changes tested? The existing `variant_get` tests build their input with `json_to_variant`, which produces `BinaryView` columns. Thus the tests with field paths and typed output now run through the new path. The new module has unit tests for top-level and nested lookups, missing fields, non-object values, an empty path, agreement with `VariantObject::get` for every field of a 300-field object, and truncated input. Benchmarks (`variant_kernels`, local run, main vs this PR, three alternating runs): | benchmark | main | this PR | change | |---|---:|---:|---:| | `variant_get_unshredded_object_path_262k_rows` | 22.21 ms | 9.99 ms | −55% | Local run on an Apple Silicon laptop. No other `variant_kernels` benchmark uses a field-name path. On real-world telemetry data with per-row metadata, we measured `variant_get` on unshredded object fields at 42–55% faster. # Are there any significant user-facing changes? No. One behavior difference: on the new path, malformed variant bytes produce an error, or a null under safe casting. The existing path panics in `Variant::new` on malformed bytes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
