andygrove commented on code in PR #4533:
URL: https://github.com/apache/datafusion-comet/pull/4533#discussion_r3436033996
##########
spark/src/main/scala/org/apache/comet/serde/arrays.scala:
##########
@@ -508,6 +508,28 @@ object CometCreateArray extends
CometExpressionSerde[CreateArray] {
return exprToProtoInternal(emptyArrayLiteral, inputs, binding)
}
+ // DataFusion's `make_array` asserts strict element-type equality in
+ // `MutableArrayData::with_capacities` and panics on a mismatch. Spark's
CreateArray is more
+ // permissive: its type coercion compares element types with `sameType`,
which ignores
+ // nullability, so children that share a surface type but differ only in
nested field
+ // nullability get no unifying cast. DataFusion tolerates container
nullability differences
+ // (an `ArrayType.containsNull` / `MapType.valueContainsNull` mismatch is
coerced), but NOT a
+ // struct field's nullability -- `array(struct(a not null), struct(a
nullable))` panics inside
+ // `make_array_inner`. Decline only those cases (i.e. children that still
differ after
+ // normalizing container nullability) so Spark's evaluator handles them.
+ //
+ // TODO: remove this decline once apache/datafusion#22366 lands; the
upstream fix widens the
+ // element type via nullability-OR-merge and casts each child before
MutableArrayData.
+ val normalizedTypes = children.map(c =>
normalizeContainerNullability(c.dataType))
Review Comment:
nit: it would be better to have this logic in `getSupportLevel` and also
implement `getIncompatibleReasons` so that this gets documented in the
compatibility guide.
Can be done later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
