[
https://issues.apache.org/jira/browse/SPARK-60362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yaoxuan Wu updated SPARK-60362:
-------------------------------
Summary: ORC pushdown of IS NULL on a nested field drops rows whose parent
struct is NULL (was: RC pushdown of IS NULL on a nested field drops rows whose
parent struct is NULL)
> ORC pushdown of IS NULL on a nested field drops rows whose parent struct is
> NULL
> --------------------------------------------------------------------------------
>
> Key: SPARK-60362
> URL: https://issues.apache.org/jira/browse/SPARK-60362
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 4.2.0, 4.1.1
> Reporter: Yaoxuan Wu
> Priority: Major
>
> {code:python}
> path = "/tmp/orc_nested_is_null"
> spark.sql("SELECT IF(id = 0, NULL, named_struct('b', id)) AS s FROM
> range(2)") \
> .coalesce(1).write.mode("overwrite").orc(path)
> spark.read.orc(path).filter("s.b IS NULL").collect()
> # expected: [Row(s=None)]
> # actual: []
> spark.conf.set("spark.sql.orc.filterPushdown", "false")
> spark.read.orc(path).filter("s.b IS NULL").collect()
> # [Row(s=None)] correct without ORC filter pushdown
> {code}
> When {{s}} is NULL, {{s.b}} is NULL as well, so the row should be returned.
> It is returned once ORC filter pushdown is turned off, and with Parquet.
> {{s.b IS NULL}} is pushed to ORC, which evaluates it on the statistics of the
> child column {{s.b}}. A row whose parent {{s}} is NULL has no entry in the
> child column, so the child's statistics show no NULLs and the row group is
> skipped.
> Reproduces on 3.5.8, 4.0.0, 4.1.1 and 4.2.0.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
