JFinis commented on PR #196:
URL: https://github.com/apache/parquet-format/pull/196#issuecomment-1625537697
> > @mapleFU @gszadovszky @pitrou @wgtmac What is your opinion on this
proposal?
>
> It's difficult to say without understanding the implications. Say a data
page contains some NaNs, what happens?
@pitrou On the write path:
* The writing library would set the `ColumnOrder` for this column to the new
option, let's call it `IEEE754TotalOrder`.
* The writing library would use IEEE754 total order for all order / sorting
related tasks. I.e., it would compute the min and max values of the page using
that total order. That order has a defined place for NaN. The writer would
*not* have special logic for NaN. It would just order everything using total
order. E.g., in case of a page containing a positive NaN, this would be chosen
as the max value, as Nan > everything else in the total order.
On the read path:
* A reading engine encountering the new `IEEE754TotalOrder` column order
would either
a) (legacy reader) not understand it and in this case not use any
statistics, as it doesn't understand the semantics of the order relation.
b) (new reader) understand it and assume that all order is in IEEE 754
total order, which again has a defined place for NaNs. Depending on the NaN
semantics of the reading engine, it would need to make sure to align the values
it sees in min/max with its own semantics. How this alignment would look like
would depend on the semantics of the engine. (I can give more detailed examples
for different engine semantics if necessary)
Ramifications:
* PRO: Due to the new column order, legacy readers are guarded. They don't
need to understand the new order. Even if they ignore the column order, if they
see NaNs in min and max they will just ignore them and assume the statistics
aren't usable. So we have two layers of protection to make sure legacy readers
don't misunderstand the ordering.
* PRO: No special fields for NaNs. No `nan_counts`, no `nan_pages`. Instead,
NaNs are just treated as defined in the total ordering.
* PRO: Simple standardized handling of floatsinstead of special handling of
NaNs. I guess this was the main point of @tustvold and @crepererum.
* PRO: Engines only need to understand total ordering and don't need any
special NaN handling code anymore (unless their semantics is different, in
which case they need to map their semantics from / to total ordering).
* CON: NaNs *will* be used in min/max bounds, even for not only-NaN pages.
This makes them less effective for filtering (as they are the widest possible
bounds) but @crepererum made a good point that this "special case for NaN" is
quite arbitrary and we could also special case INT_MAX for integer columns,
e.g.. I see the argument that keeping the architecture simple might be
preferrable. Also NaNs are not widely used, so this will not be determinental
to many data sets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
