dayan02dev opened a new pull request, #12246: URL: https://github.com/apache/arrow-rs/pull/12246
# Which issue does this PR close? - Closes #11946 - Part of #10546 # Rationale for this change `cast_to_variant` returned `Ok` for duration/interval types (directly or nested) when the input was empty or all null, and the unsupported-type error as soon as one row held a value. There are two causes: 1. The all-null fast path in `cast_to_variant_with_options` returned before `make_arrow_to_variant_row_builder`, which is the only place types are checked. 2. `UnionArrowToVariantBuilder::new` looped over the per-row `type_ids()` buffer, so it only built (and type-checked) the children some row selects, and rebuilt that child's builder once per row. Moving the type check before the shortcut, as the issue suggests, fixes the first cause but not the second: an empty union with a duration child, or one whose rows never select that child, is still accepted. So the union loop also changes. # What changes are included in this PR? - Build the row builder before the all-null fast path, so the existing recursive type check runs first. Supported types still take the bulk `append_nulls` fast path. - Create one union child builder per declared field (`union_array.fields().iter()`) instead of per row. This matches the comment above the loop and the behaviour before #8299. #12211 also edits the body of the same union loop, so one of the two will need a small rebase. # Are these changes tested? Yes. `test_cast_to_variant_duration_or_interval_errors_empty_or_all_null` covers the seven duration/interval types as a scalar, list, struct and sparse union, both empty and all null, plus supported controls. Without the fix it fails with `List(Duration(s)) with 0 rows: Ok(())`; with only the fast-path reorder it still fails on the empty union case. # Are there any significant user-facing changes? - Empty or all-null arrays of unsupported types (including nested ones), and unions that declare an unsupported child, now return the same `InvalidArgumentError` as populated arrays instead of `Ok`. No signatures change. - All-null map and dictionary inputs now pay builder construction (for maps, casting the keys to Utf8; for dictionaries, normalising the keys). Because a map slice's keys are the parent's whole entries array, an all-null 1000-row slice of a 1M-entry map goes from ~0.01 ms to ~17 ms, the same as a populated slice of that map already costs. Other all-null types show no measurable change. - Valid unions convert ~1.8× faster, since child builders are no longer rebuilt per row. If you'd rather keep the fast path cost-free for maps, the alternatives are narrowing the map key cast to the offsets range, or a `DataType`-only recursive check before the shortcut. # AI usage disclosure The patch, test, verification and the benchmark numbers above were produced with an AI coding agent (Claude Code), and a separate agent session re-checked them before submission. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
