dayan02dev opened a new pull request, #12246:
URL: https://github.com/apache/arrow-rs/pull/12246

   # Which issue does this PR close?
   
   - Closes #11946
   - Part of #10546
   
   # Rationale for this change
   
   `cast_to_variant` returned `Ok` for duration/interval types (directly or 
nested) when the input was empty or all null, and the unsupported-type error as 
soon as one row held a value. There are two causes:
   
   1. The all-null fast path in `cast_to_variant_with_options` returned before 
`make_arrow_to_variant_row_builder`, which is the only place types are checked.
   2. `UnionArrowToVariantBuilder::new` looped over the per-row `type_ids()` 
buffer, so it only built (and type-checked) the children some row selects, and 
rebuilt that child's builder once per row.
   
   Moving the type check before the shortcut, as the issue suggests, fixes the 
first cause but not the second: an empty union with a duration child, or one 
whose rows never select that child, is still accepted. So the union loop also 
changes.
   
   # What changes are included in this PR?
   
   - Build the row builder before the all-null fast path, so the existing 
recursive type check runs first. Supported types still take the bulk 
`append_nulls` fast path.
   - Create one union child builder per declared field 
(`union_array.fields().iter()`) instead of per row. This matches the comment 
above the loop and the behaviour before #8299.
   
   #12211 also edits the body of the same union loop, so one of the two will 
need a small rebase.
   
   # Are these changes tested?
   
   Yes. `test_cast_to_variant_duration_or_interval_errors_empty_or_all_null` 
covers the seven duration/interval types as a scalar, list, struct and sparse 
union, both empty and all null, plus supported controls. Without the fix it 
fails with `List(Duration(s)) with 0 rows: Ok(())`; with only the fast-path 
reorder it still fails on the empty union case.
   
   # Are there any significant user-facing changes?
   
   - Empty or all-null arrays of unsupported types (including nested ones), and 
unions that declare an unsupported child, now return the same 
`InvalidArgumentError` as populated arrays instead of `Ok`. No signatures 
change.
   - All-null map and dictionary inputs now pay builder construction (for maps, 
casting the keys to Utf8; for dictionaries, normalising the keys). Because a 
map slice's keys are the parent's whole entries array, an all-null 1000-row 
slice of a 1M-entry map goes from ~0.01 ms to ~17 ms, the same as a populated 
slice of that map already costs. Other all-null types show no measurable change.
   - Valid unions convert ~1.8× faster, since child builders are no longer 
rebuilt per row.
   
   If you'd rather keep the fast path cost-free for maps, the alternatives are 
narrowing the map key cast to the offsets range, or a `DataType`-only recursive 
check before the shortcut.
   
   # AI usage disclosure
   
   The patch, test, verification and the benchmark numbers above were produced 
with an AI coding agent (Claude Code), and a separate agent session re-checked 
them before submission.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to