[
https://issues.apache.org/jira/browse/SPARK-60322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60322:
-----------------------------------
Labels: correctness pull-request-available (was: correctness)
> Parquet and ORC filter pushdown drops rows for comparisons with 0.0 or -0.0
> ---------------------------------------------------------------------------
>
> Key: SPARK-60322
> URL: https://issues.apache.org/jira/browse/SPARK-60322
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.3, 4.2.0, 4.0.3, 4.3.0, 5.0.0, 4.4.0
> Reporter: sepuri sai krishna
> Priority: Major
> Labels: correctness, pull-request-available
>
> Filter pushdown to Parquet and ORC skips rows when a FLOAT or DOUBLE column
> is compared with a zero and the data holds the other zero. There is no error,
> the rows are just missing. With the default configuration:
> {code:sql}
> CREATE TABLE p USING parquet AS SELECT id, -0.0D * id AS f FROM range(0,
> 1000, 1, 1);
> SELECT count(*) FROM p WHERE f = 0; -- 0, expected 1000
> CREATE TABLE o USING orc AS SELECT id, -0.0D * id AS f FROM range(1, 1000, 1,
> 1);
> SELECT count(*) FROM o WHERE f = 0; -- 0, expected 999
> SELECT count(*) FROM o WHERE f >= 0; -- 0, expected 999
> SET spark.sql.parquet.filterPushdown=false;
> SET spark.sql.orc.filterPushdown=false;
> SELECT count(*) FROM p WHERE f = 0; -- 1000
> SELECT count(*) FROM o WHERE f = 0; -- 999
> {code}
> Same results on 3.5.3, 4.0.3 and 4.2.0 (official binaries) and on master.
> Spark treats 0.0 and -0.0 as equal ({{0.0D = -0.0D}} is true), but
> {{ParquetFilters}} and {{OrcFilters}} push the literal down unchanged, and
> the formats tell the two zeros apart:
> * Parquet dictionary filter and bloom filter: a dictionary or bloom filter
> that holds only -0.0 does not contain 0.0, so the row group is skipped. This
> is what drops the Parquet rows above.
> * ORC statistics: ORC compares with {{Double.compareTo}}, so a stripe or row
> group whose min and max are -0.0 is skipped for {{f = 0.0}} or {{f >= 0.0}},
> and one whose min and max are 0.0 for {{f = -0.0}} or {{f <= -0.0}}.
> * Parquet statistics and column index: Parquet 1.17 and earlier write -0.0 as
> the min and 0.0 as the max of a column chunk of zeros, so they do not skip
> anything. Parquet 1.18 (master only, SPARK-58869) writes FLOAT and DOUBLE
> columns with the IEEE 754 total order and exact bounds, so on master the
> statistics and the column index skip rows as well, for files written with
> 1.18.
> Affected predicates, with z = 0.0 or -0.0: {{f = z}}, {{f <=> z}}, {{f IN (z,
> ...)}}, {{f <= -0.0}}, {{f >= 0.0}}, and BETWEEN / OR built from them. A
> pushed {{Not}} of {{f < z}}, {{f > z}} or {{f != z}} is wrong as well, since
> it negates a comparison that keeps more values than the Spark predicate. The
> optimizer usually rewrites such a {{NOT}} away, but not when that rule is
> excluded.
> Both FLOAT and DOUBLE, DSv1 and DSv2, vectorized and non-vectorized readers.
> -0.0 is easy to get: {{-x}} or {{x * -1}} for x = 0.0, and data written by
> other engines (pandas/numpy, Arrow, C++ writers) keeps the sign.
> Workaround: {{spark.sql.parquet.filterPushdown=false}} /
> {{spark.sql.orc.filterPushdown=false}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
