sunchao commented on code in PR #5407:
URL: https://github.com/apache/datafusion-comet/pull/5407#discussion_r3840982517
##########
native/core/src/parquet/cast_column.rs:
##########
@@ -176,6 +186,285 @@ fn cast_timestamp_micros_to_millis_scalar(
ScalarValue::TimestampMillisecond(new_val, target_tz)
}
+fn normalize_variant_array(
+ array: &ArrayRef,
+ target_field: &FieldRef,
+) -> DataFusionResult<ArrayRef> {
+ let DataType::Struct(fields) = target_field.data_type() else {
+ return Err(DataFusionError::Execution(
+ "Variant extension field must use Struct storage".to_string(),
+ ));
+ };
+ if fields.len() != 2
+ || fields[0].name() != "value"
+ || fields[1].name() != "metadata"
+ || fields
+ .iter()
+ .any(|field| field.data_type() != &DataType::Binary)
+ {
+ return Err(DataFusionError::Execution(
+ "Variant output must contain Binary children [value,
metadata]".to_string(),
+ ));
+ }
+
+ let array = decode_variant_metadata_dictionary(array)?;
+ let variant =
prepare_variant_for_unshredding(&VariantArray::try_new(array.as_ref())?)?;
Review Comment:
[P2] Widen unsigned shredded Variant children before normalization
Please widen unsigned `typed_value` children before constructing
`VariantArray`, or keep these reads on Spark. With
`spark.sql.variant.pushVariantIntoScan=false` and
`spark.sql.variant.allowReadingShredded=true`, Spark 4.0.4/4.1.3's vectorized
reader accepts ordinary Parquet files whose Variant `typed_value` is `INT32
(INTEGER(8,false))`, `(16,false)`, or `(32,false)`, including values `255`,
`65535`, and `4294967295`. I reproduced all three failures through the
exact-head `CometNativeScan`: Arrow/Parquet 58.4 restores
`UInt8`/`UInt16`/`UInt32`, which this constructor rejects with `Illegal
shredded value type: UInt8` (or 16/32). These files have no embedded
`ARROW:schema`, and the new top-level Variant gate admits them. Widening only
that child to `Int16`/`Int32`/`Int64` makes the current normalizer accept the
same rows. An unsigned upper-bound native-read regression would cover this
boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
