Carl Sverre created SPARK-31403:
-----------------------------------
Summary: TreeNode asCode function incorrectly handles null literals
Key: SPARK-31403
URL: https://issues.apache.org/jira/browse/SPARK-31403
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.4
Reporter: Carl Sverre
In the TreeNode code in Catalyst the asCode function incorrectly handles null
literals. When it tries to render a null literal it will match {{null}} using
the third case expression and try to call {{null.toString}} which will raise a
NullPointerException.
I verified this bug exists in Spark 2.4.4 and the same code appears to be in
master:
[https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala#L707]
The fix seems trivial - add an explicit case for null.
One way to reproduce this is via:
{code:java}
val plan =
spark
.sql("select if(isnull(id), null, 2) from testdb_jdbc.users")
.queryExecution
.optimizedPlan
println(plan.asInstanceOf[Project].projectList.head.asCode) {code}
However any other way which generates a Literal with the value null will cause
the issue.
In this case the above SparkSQL will generate the literal: {{Literal(null,
IntegerType)}} for the "trueValue" of the if statement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
