Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/9770#discussion_r45104293
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1063,6 +1065,29 @@ class Analyzer(
Project(p.output, newPlan.withNewChildren(newChild :: Nil))
}
}
+
+ /**
+ * Correctly handle null primitive inputs for UDF by adding extra [[If]]
expression to do the
+ * null check. When user defines a UDF with primitive parameters, there
is no way to tell if the
+ * primitive parameter is null or not, so here we assume the primitive
input is null-propagatable
+ * and we should return null if the input is null.
+ */
+ object HandleNullInputsForUDF extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan
resolveOperators {
+ case p if !p.resolved => p // Skip unresolved nodes.
+
+ case plan => plan transformExpressionsUp {
+
+ case udf @ ScalaUDF(func, _, inputs, _) =>
+ val parameterTypes = ScalaReflection.getParameterTypes(func)
+ assert(parameterTypes.length == inputs.length)
+
+
parameterTypes.zip(inputs).filter(_._1.isPrimitive).map(_._2).foldLeft(udf:
Expression) {
+ case (result, input) => If(IsNull(input), Literal.create(null,
udf.dataType), result)
--- End diff --
I think this would be a lot easier to read in the query plan if you created
a single `If` with `Or`s.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
