Kimahriman commented on code in PR #793:
URL: https://github.com/apache/datafusion-comet/pull/793#discussion_r1711318695
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -2348,6 +2348,27 @@ object QueryPlanSerde extends Logging with
ShimQueryPlanSerde with CometExprShim
.build()
}
+ // datafusion's make_array only supports nullable element types
+ case array @ CreateArray(children, _) if array.dataType.containsNull =>
+ val childExprs = children.map(exprToProto(_, inputs, binding))
+ val dataType = serializeDataType(array.dataType)
Review Comment:
Hmm looking a little more there's a chance I might be wrong. I think
`return_type` is only used to infer the return type if it's not specified? I
think the real issue is that `invoke` (which uses `fn array_array`) under the
hood, returns an `Array` with an always nullable element. But here we have the
`ArrayRef`'s which have an `is_nullable` attribute, so it might be possible.
I'll try to test that out to see if that's the case. The actual error you get
is:
```
Cause: org.apache.comet.CometNativeException: Invalid argument error: column
types must match schema types, expected List(Field { name: "item", data_type:
Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }) but
found List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0,
dict_is_ordered: false, metadata: {} }) at column index 0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
