chenjian2664 opened a new pull request, #10826:
URL: https://github.com/apache/paimon/pull/10826
### Purpose
`SparkFilterConverter` already converts `EqualTo(col, NaN)` into an `IsNaN`
predicate (#8060). An `Equal` predicate with a NaN literal can prune data files
whose min/max statistics exclude NaN, and the matching rows are then silently
dropped. The `EqualNullSafe` branch did not get the same handling, so a filter
such as `col <=> CAST('NaN' AS DOUBLE)` still became `Equal(col, NaN)`. This
converter is still used where Spark pushes down V1 filters, for example scan
filters on Spark 3.2 and runtime filtering on Spark 3.2 and 3.3.
This change maps `EqualNullSafe(col, NaN)` to `IsNaN(col)` for float and
double values, the same as the `EqualTo` branch and `SparkV2FilterConverter`.
Null and non-NaN values keep their current conversion.
### Tests
Added `SparkFilterConverterTest#testEqualNullSafeNaN`, which covers
`EqualNullSafe` with a NaN literal on float and double columns converting to
`IsNaN`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
