[
https://issues.apache.org/jira/browse/SPARK-59605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59605:
-----------------------------------
Labels: pull-request-available (was: )
> ORC filter pushdown fails on an `IN` list containing `NULL`
> -----------------------------------------------------------
>
> Key: SPARK-59605
> URL: https://issues.apache.org/jira/browse/SPARK-59605
> Project: Spark
> Issue Type: Bug
> Components: Bug, Optimizer
> Affects Versions: 5.0.0
> Reporter: Yibo Dong
> Priority: Major
> Labels: pull-request-available
>
> h2. What happened
> ORC filter pushdown fails when an {{IN}} predicate contains a {{NULL}}
> literal.
> The query executes correctly when {{spark.sql.orc.filterPushdown}} is
> disabled, but fails with a {{NullPointerException}} when filter pushdown is
> enabled.
> h2. How to reproduce
> Run the following SQL:
> {code:sql}
> DROP TABLE IF EXISTS T;
> CREATE TABLE T USING ORC AS
> SELECT id, label
> FROM VALUES
> (NULL, 'null'),
> (1, 'a'),
> (2, 'b'),
> (3, 'c') AS t(id, label);
> SELECT id, label
> FROM T
> WHERE id IN (1, 3, NULL)
> ORDER BY id, label;
> {code}
> Run it with ORC filter pushdown enabled:
> {code:bash}
> spark-sql
> --master 'local[2]'
> --conf spark.ui.enabled=false
> --conf spark.sql.adaptive.enabled=false
> --conf spark.sql.shuffle.partitions=1
> --conf spark.sql.orc.filterPushdown=true
> -f repro.sql
> {code}
> Then run the same query with ORC filter pushdown disabled:
> {code:bash}
> spark-sql
> --master 'local[2]'
> --conf spark.ui.enabled=false
> --conf spark.sql.adaptive.enabled=false
> --conf spark.sql.shuffle.partitions=1
> --conf spark.sql.orc.filterPushdown=false
> -f repro.sql
> {code}
> h2. Expected result
> {{id IN (1, 3, NULL)}} should match the rows with {{id = 1}} and {{id = 3}}.
> Both configurations should return:
> {code}
> 1 a
> 3 c
> {code}
> h2. Actual result
> With {{spark.sql.orc.filterPushdown=false}}, Spark returns the expected rows:
> {code}
> 1 a
> 3 c
> {code}
> With {{spark.sql.orc.filterPushdown=true}}, the query fails while
> constructing the ORC filter:
> {code}
> org.apache.spark.SparkException: [FAILED_READ_FILE.NO_HINT]
> Encountered error while reading file ...
> Caused by: java.lang.NullPointerException:
> Cannot invoke "java.lang.Number.longValue()" because "value" is null
> at
> org.apache.spark.sql.execution.datasources.orc.OrcFilters$.castLiteralValue(OrcFilters.scala:167)
> at
> org.apache.spark.sql.execution.datasources.orc.OrcFilters$.buildLeafSearchArgument(OrcFilters.scala:284)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
