Kimahriman commented on code in PR #793:
URL: https://github.com/apache/datafusion-comet/pull/793#discussion_r1709318923
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -2348,6 +2348,27 @@ object QueryPlanSerde extends Logging with
ShimQueryPlanSerde with CometExprShim
.build()
}
+ // datafusion's make_array only supports nullable element types
+ case array @ CreateArray(children, _) if array.dataType.containsNull =>
+ val childExprs = children.map(exprToProto(_, inputs, binding))
+ val dataType = serializeDataType(array.dataType)
Review Comment:
I could "fake" the datatype here when the element isn't nullable and tell
DataFusion it is to get it to work, but I wasn't sure if that would have
unintended downstream consequences. I can try to update DataFusion at some
point to support non-nullable elements if all children are non-nullable.
##########
native/core/src/execution/datafusion/planner.rs:
##########
@@ -623,6 +624,21 @@ impl PhysicalPlanner {
let child = self.create_expr(expr.child.as_ref().unwrap(),
input_schema.clone())?;
Ok(Arc::new(GetStructField::new(child, expr.ordinal as usize)))
}
+ ExprStruct::CreateArray(expr) => {
+ let values = expr
+ .values
+ .iter()
+ .map(|expr| self.create_expr(expr, input_schema.clone()))
+ .collect::<Result<Vec<_>, _>>()?;
+ let data_type =
to_arrow_datatype(expr.datatype.as_ref().unwrap());
+
+ Ok(Arc::new(ScalarFunctionExpr::new(
+ "make_array",
+ make_array_udf(),
+ values,
+ data_type,
+ )))
Review Comment:
I'm not sure if this is the right/best way to make use of a ScalarUDF in
DataFusion, but it compiled and seemed to work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
