[
https://issues.apache.org/jira/browse/FLINK-40927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124896#comment-18124896
]
Yahya Kisana commented on FLINK-40927:
--------------------------------------
[~Sergey Nuyanzin] Because some places in Flink follow IEEE 754, some follow
NaN as a value.
ScalarOperatorGens.scala:622-627 follows IEEE: NaN == NaN -> False
RexSimplify.java:1003-1018 follow NaN as a value: NaN == NaN -> True (assumes a
value always equals itself, so NaN behaves as if NaN = NaN → TRUE when both
sides are the same expression.)
My point, should Flink stick to one.
This causes issues like this: https://issues.apache.org/jira/browse/FLINK-40923
> Inconsistent NaN semantics for FLOAT/DOUBLE across planner and runtime
> -----------------------------------------------------------------------
>
> Key: FLINK-40927
> URL: https://issues.apache.org/jira/browse/FLINK-40927
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
> Reporter: Yahya Kisana
> Priority: Major
>
> Flink has no single definition of how NaN behaves for FLOAT/DOUBLE. Different
> parts of the planner and runtime treat it differently, so results can depend
> on the plan rather than the data.
> ||Area||NaN behaviour||Code||
> |Comparisons in filters/projections|IEEE 754: {{NaN = NaN}} is
> FALSE|[ScalarOperatorGens.scala:623-627|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/calls/ScalarOperatorGens.scala#L623-L627]|
> |Planner simplification (Calcite)|Assumes values equal themselves and are
> totally ordered|[RexSimplify.java:
> 1004-1019)|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/calcite/rex/RexSimplify.java#L1004-L1019]|
> |Sort comparator (ORDER BY, TopN)|NaN compares equal to every value (not a
> valid
> ordering)|[GenerateUtils.scala:638-640|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/GenerateUtils.scala#L638-L640]|
> |Batch sort normalized keys|NaN sorts above
> +Infinity|[SortUtil.java:102-115|https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/sort/SortUtil.java#L102-L105]|
> |RecordEqualiser|Binary rows: NaN = NaN; other rows: NaN !=
> NaN|[EqualiserCodeGenerator.scala:72-74 vs
> 159-160|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/EqualiserCodeGenerator.scala]|
> |GROUP BY / DISTINCT / join keys|Byte comparison: NaNs with the same bits are
> equal|[BinarySection.java:65-75|https://github.com/apache/flink/blob/master/flink-table/flink-table-common/src/main/java/org/apache/flink/table/data/binary/BinarySection.java#L65-L75]|
> |NaN constants|Cannot be represented
> (BigDecimal)|[ValueLiteralExpression.java:216|https://github.com/apache/flink/blob/master/flink-table/flink-table-common/src/main/java/org/apache/flink/table/expressions/ValueLiteralExpression.java#L216]|
> Other engines take one of two positions:
> * IEEE 754 everywhere ({{NaN = NaN}} is FALSE): Trino and Impala (which
> disables affected simplifications).
> * NaN is a normal value, equal to itself and greater than all other numbers:
> Spark/Databricks, PostgreSQL, DuckDB. This makes the planner's assumptions
> valid and gives sorting, grouping and joins one consistent order.
> This was found while working on
> [FLINK-40923|https://issues.apache.org/jira/browse/FLINK-40923]
> Needs further discussion
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
