Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/23239#discussion_r239507673
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeWriter.java
---
@@ -198,11 +198,45 @@ protected final void writeLong(long offset, long
value) {
Platform.putLong(getBuffer(), offset, value);
}
+ // We need to take care of NaN and -0.0 in several places:
+ // 1. When compare values, different NaNs should be treated as same,
`-0.0` and `0.0` should be
+ // treated as same.
+ // 2. In range partitioner, different NaNs should belong to the same
partition, -0.0 and 0.0
--- End diff --
It turns out this is not a problem. The doc of `RangePartitioning` is
misleading. I'm updating the doc at https://github.com/apache/spark/pull/23249
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
