sdf-jkl opened a new issue, #11946: URL: https://github.com/apache/arrow-rs/issues/11946
## Describe the bug `cast_to_variant` succeeds for empty and all-null arrays of unsupported duration and interval types, including when nested in lists or structs. The same source types fail when the arrays contain a non-null duration or interval. Type acceptance therefore depends on batch contents. Found while implementing Spark-compatible Variant conversions in [datafusion-variant](https://github.com/datafusion-contrib/datafusion-variant). This prevents consistent rejection of unsupported SQL casts and makes empty-array probes unreliable for determining kernel support. ## To Reproduce Executed with `arrow` and `parquet-variant-compute` version `60.0.0`: ```rust use std::sync::Arc; use arrow::array::{new_empty_array, new_null_array, ArrayRef, IntervalYearMonthArray, ListArray}; use arrow::buffer::OffsetBuffer; use arrow::datatypes::{DataType, Field, IntervalUnit}; use parquet_variant_compute::cast_to_variant; fn main() { let dt = DataType::Interval(IntervalUnit::YearMonth); let values: ArrayRef = Arc::new(IntervalYearMonthArray::from(vec![1])); let child = Arc::new(Field::new("item", dt.clone(), true)); let list_type = DataType::List(child.clone()); let list: ArrayRef = Arc::new(ListArray::new(child, OffsetBuffer::new(vec![0, 1].into()), values.clone(), None)); for (name, array) in [ ("interval empty", new_empty_array(&dt)), ("interval all-null", new_null_array(&dt, 1)), ("interval non-null", values), ("list<interval> empty", new_empty_array(&list_type)), ("list<interval> all-null", new_null_array(&list_type, 1)), ("list<interval> non-null", list), ] { match cast_to_variant(array.as_ref()) { Ok(_) => println!("{name}: Ok"), Err(e) => println!("{name}: {e}"), } } } ``` Observed results: | Input | Interval | List of interval | | --- | --- | --- | | Empty array | Success | Success | | One null row | Success | Success | | One non-null row containing an interval | Unsupported-type error | Unsupported-type error | Both failures report: `Casting duration/interval types to Variant is not supported. The Variant format does not define duration/interval types.` Additional probes on 60.0.0 reproduced the same behavior for all four duration units (`Second`, `Millisecond`, `Microsecond`, `Nanosecond`) and all three interval units (`YearMonth`, `DayTime`, `MonthDayNano`). Each was checked as a scalar array, a list element, and a struct field: empty and all-null outer arrays succeeded, while populated arrays failed. ## Expected behavior Reject unsupported source types consistently, including nested unsupported types, regardless of batch length or null contents. Check recursive type support before the empty/all-null shortcut. Supported types should retain their empty/all-null behavior, and `DataType::Null` should remain supported. Value-dependent conversion failures, such as overflow, should continue to follow `CastOptions`; these are separate from unsupported source types. Regression tests should cover empty, all-null, and populated arrays for both scalar and nested unsupported types, with supported-type controls. ## Additional context On current `main` at `5746b443f7`, the [empty/all-null shortcut](https://github.com/apache/arrow-rs/blob/5746b443f70b4e55676be59cea317501a94ec810/parquet-variant-compute/src/cast_to_variant.rs#L57-L70) still precedes the [private builder dispatch](https://github.com/apache/arrow-rs/blob/5746b443f70b4e55676be59cea317501a94ec810/parquet-variant-compute/src/arrow_to_variant.rs#L150-L155), which [rejects interval and duration types](https://github.com/apache/arrow-rs/blob/5746b443f70b4e55676be59cea317501a94ec810/parquet-variant-compute/src/arrow_to_variant.rs#L227-L233). A useful follow-up would be to expose the shared recursive check for downstream planning, analogous to Arrow's ordinary [`can_cast_types`](https://github.com/apache/arrow-rs/blob/5746b443f70b4e55676be59cea317501a94ec810/arrow-cast/src/cast/mod.rs#L252). For example, a proposed `validate_cast_to_variant(&Field) -> Result<(), ArrowError>` could let planners check support without arrays or a duplicated type whitelist. A public API is not required to fix this bug; engine-specific restrictions would remain downstream. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
