Yaoxuan Wu created SPARK-60916:
----------------------------------
Summary: Parquet filter pushdown of IN (..., NULL) on a BOOLEAN
column drops matching rows when the list is longer than
spark.sql.parquet.pushdown.inFilterThreshold
Key: SPARK-60916
URL: https://issues.apache.org/jira/browse/SPARK-60916
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1, 4.2.0, 3.5.8, 4.0.0
Reporter: Yaoxuan Wu
{code:python}
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
path = "/tmp/bool_in_null"
spark.sql("SELECT * FROM VALUES (true), (false) AS
t(b)").coalesce(1).write.mode("overwrite").parquet(path)
spark.sql("SELECT CAST(NULL AS BOOLEAN) AS
b").write.mode("append").parquet(path)
spark.read.parquet(path).createOrReplaceTempView("t")
q = "SELECT b FROM t WHERE b IN (true, NULL)"
spark.sql(q).show() # one row: true
spark.conf.set("spark.sql.parquet.pushdown.inFilterThreshold", "1")
spark.sql(q).show() # expected one row: true; actual: no row
{code}
Same on 3.5.8, 4.0.0, 4.1.1 and 4.2.0. {{b IN (true, false, NULL)}} with
threshold 2 also returns no row. With
{{spark.sql.parquet.filterPushdown=false}}, or for an INT column ({{x IN (1,
NULL)}}), the result is right.
Cause: {{ParquetFilters.createFilterHelper}}, {{case sources.In}} (v4.2.0 lines
798-818, unchanged on master). When the IN list has more values than
{{inFilterThreshold}} and contains NULL, the pushed filter is an OR of an
equality with NULL and a Parquet IN predicate on the other values:
{code:scala}
if (values.contains(null)) {
Seq(makeEq.lift(fieldType).map(_(fieldNames, null)),
makeInPredicate.lift(fieldType).map(_(fieldNames, values.filter(_ != null)))
).flatten.reduceLeftOption(FilterApi.or)
{code}
{{makeEq}} has a {{ParquetBooleanType}} case, but {{makeInPredicate}} has none
(it is the only type {{makeEq}} supports and {{makeInPredicate}} does not). For
a BOOLEAN column the second element is {{None}}, {{flatten}} drops it, and the
pushed filter becomes {{b = NULL}} alone, which is stricter than the original
predicate; Parquet then skips every row group without NULLs. Pushing nothing
when either part cannot be built, or adding a BOOLEAN case to
{{makeInPredicate}}, would fix it. The code was added in SPARK-39312 (3.4.0).
The default threshold (10) is not affected: {{OptimizeIn}} removes duplicate
values from the IN list ({{ExpressionSet(list)}}), and a BOOLEAN list has at
most three distinct values, so this needs {{inFilterThreshold}} <= 2.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
