[
https://issues.apache.org/jira/browse/SPARK-58482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132720#comment-18132720
]
Max Gekk commented on SPARK-58482:
----------------------------------
The defect is wider than a NaN literal: any range filter whose literal lies
above a batch's
largest non-NaN value prunes that batch, and with it the batch's NaN rows,
which Spark's ordering
places above every value. So a plain {{WHERE v > 5.0}} on a cached table loses
NaN rows.
Reproduced on master (DefaultCachedBatchSerializer, partition pruning on, one
batch):
{code:scala}
spark.range(1).selectExpr(
"inline(array(struct(1, 1.5D, 1.5F), struct(2, CAST('NaN' AS DOUBLE),
CAST('NaN' AS FLOAT)), " +
"struct(3, 0.0D, 0.0F)))").toDF("id", "v",
"f").repartition(1).createOrReplaceTempView("n")
spark.sql("SELECT id FROM n WHERE v > 5.0D").show() // id 2
spark.catalog.cacheTable("n")
spark.sql("SELECT id FROM n WHERE v > 5.0D").show() // no rows
{code}
||predicate||uncached||cached||
|v = NaN|2|none|
|v <=> NaN|2|none|
|v >= NaN|2|none|
|v > 5.0, 5.0 < v, v >= 2.0|2|none|
|v IN (NaN, 7.0)|2|none|
|f = NaN, f > 5.0 (FLOAT)|2|none|
|v > 0.0 on a batch holding only NaN|the row|none|
|isnan(v), v < 1.0|same|same|
By the code history the mismatch dates from SPARK-8797 (Spark 1.5, July 2015),
which gave Catalyst
its NaN ordering after SPARK-2961 (Spark 1.2) had added batch pruning on
IEEE-gathered bounds; the
affected versions likely go back to 1.5.
The fix in https://github.com/apache/spark/pull/57701 makes NaN the batch's
upper bound, which
keeps the batch for all of the cases above. Its end-to-end test covers only {{v
= NaN}}; a range
case such as {{v > 5.0}} and a NaN-only batch would pin the wider behaviour,
along with the FLOAT
test the review already asks for.
> Filtering a cached table for NaN values returns incorrect results:
> cached-batch pruning compares under a different ordering than the statistics
> were built with
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58482
> URL: https://issues.apache.org/jira/browse/SPARK-58482
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Josh Rosen
> Priority: Major
> Labels: correctness, pull-request-available
>
> CACHE TABLE causes the following query to return incorrect results:
> {code:sql}
> CREATE TABLE n(id INT, v DOUBLE) USING PARQUET;
> INSERT INTO n VALUES (1, 1.5), (2, CAST('NaN' AS DOUBLE)), (3, 0.0);
> SELECT id FROM n WHERE v = CAST('NaN' AS DOUBLE); -- 1 row, correct
> CACHE TABLE n;
> SELECT id FROM n WHERE v = CAST('NaN' AS DOUBLE); -- 0 rows
> {code}
> Caching the table changes the answer. Spark's equality treats NaN as equal to
> itself, so the
> correct result is 1 row in both cases.
> The batch statistics are gathered with IEEE comparison.
> In {{{}DoubleColumnStats.gatherValueStats
> ([src|https://github.com/apache/spark/blob/4dcdd4dba544210898180a410b8b30e0b85157cb/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnStats.scala#L247-L252])
> the bounds are updated with \{{{}if (value > upper) upper = value{}}}, and
> every comparison against NaN is false, so NaN never widens the maximum. The
> batch containing NaN therefore reports bounds drawn only from its non-NaN
> values.
> The pruning test then compares under Catalyst's total ordering, where NaN
> sorts above all other values.
> {{SimpleMetricsCachedBatchSerializer.buildFilter}} evaluates
> {{{}lowerBound <= lit && lit <= upperBound{}}}, the literal NaN is above the
> recorded maximum, and the batch is skipped. The row is not returned.
> {{FloatColumnStats}} has the same shape and the same defect.
> Related: SPARK-24934 fixed the same class of problem for partition pruning,
> where min/max
> statistics were used for types whose comparison semantics did not match. This
> is the columnar cache instance of it.
> Suggested direction: gather the bounds using the same ordering the pruning
> test uses, or record a NaN-present flag and skip bounds pruning for a NaN
> literal. Whichever is chosen, producer and consumer need to agree on the
> comparator, which is the actual invariant being broken here.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
