Thomas Graves created SPARK-33544:
-------------------------------------
Summary: explode should not filter when used with CreateArray
Key: SPARK-33544
URL: https://issues.apache.org/jira/browse/SPARK-33544
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.1.0
Reporter: Thomas Graves
https://issues.apache.org/jira/browse/SPARK-32295 added in an optimization to
insert a filter for not null and size > 0 when using inner explode/inline. This
is fine in most cases but the extra filter is not needed if the explode is with
a create array and not using Literals (it already handles LIterals). When this
happens you know that the values aren't null and it has a size. It already
handles the empty array.
for instance:
val df = someDF.selectExpr("number", "explode(array(word, col3))")
So in this case we shouldn't be inserting the extra Filter and that filter can
get pushed down into like a parquet reader as well. This is just causing extra
overhead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
