rdblue commented on code in PR #514:
URL: https://github.com/apache/parquet-format/pull/514#discussion_r3269900728
##########
src/main/thrift/parquet.thrift:
##########
@@ -1111,23 +1122,100 @@ union ColumnOrder {
* 64-bit signed integer (nanos)
* See https://github.com/apache/parquet-format/issues/502 for more
details
*
- * (*) Because the sorting order is not specified properly for floating
- * point values (relations vs. total ordering) the following
+ * (*) Because TYPE_ORDER is ambiguous for floating point types due to
+ * underspecified handling of NaN and -0/+0, it is recommended that
writers
+ * use IEEE_754_TOTAL_ORDER for these types.
+ *
+ * If TYPE_ORDER is used for floating point types, then the following
* compatibility rules should be applied when reading statistics:
* - If the min is a NaN, it should be ignored.
* - If the max is a NaN, it should be ignored.
+ * - If the nan_count field is set, a reader can compute
+ * nan_count + null_count == num_values to deduce whether all non-null
+ * values are NaN.
* - If the min is +0, the row group may contain -0 values as well.
* - If the max is -0, the row group may contain +0 values as well.
* - When looking for NaN values, min and max should be ignored.
+ * If the nan_count field is set, it can be used to check whether
+ * NaNs are present.
*
- * When writing statistics the following rules should be followed:
- * - NaNs should not be written to min or max statistics fields.
+ * When writing statistics for columns with TYPE_ORDER order, then
+ * following rules must be followed:
+ * - Always set the nan_count field for floating point types, even if
+ * it is zero.
Review Comment:
In order for this to be reliable, the presence of `nan_count` when the order
is set to `TYPE_ORDER` must mean that the writer guarantees NaN values were
removed before comparison to determine the lower and upper bounds, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
