orlp commented on PR #221:
URL: https://github.com/apache/parquet-format/pull/221#issuecomment-2931376077
This proposal has a **huge** problem which `nan_counts` does not have: NaNs
in the dataset poisons min/max pruning for queries which ignore nans. If the
dataset contains both a signed positive NaN and a signed negative NaN the
statistics are entirely poisoned.
I think the best solution is to add a `nan_counts` fields along with a new
ordering. This ordering is *different* for min and max:
min: smallest non-NaN value, or NaN otherwise (equivalent to NaN last)
max: largerst non-NaN value, or NaN otherwise (equivalent to NaN first)
The all-NaN page will then have `(min, max) = (NaN, NaN)`, otherwise there
will be no NaN.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
