On Fri, Feb 16, 2018 at 9:15 AM, Tim Armstrong <tarmstr...@cloudera.com> wrote:
> I don't see a major benefit to a temporary solution. The files are already > out there and we need to implement a fix on the read path regardless. If we > keep writing the stats there's at least some information contained in the > stats that readers can make use of, if they want to implement the required > logic. > > Dropping stats if an NaN is encountered also doesn't really address the > other side of the problem - an absence of a NaN in the stats doesn't imply > an absence of a NaN in the data, so the reader can't do anything useful > with the stats anyway unless it's NaN-aware. > The writer solution is to only write stats if the data does not contain special values (common case). > > On Fri, Feb 16, 2018 at 9:03 AM, Alexander Behm <alex.b...@cloudera.com> > wrote: > > > I hope the common cases is that data files do not contain these special > > float values. As the simplest solution, how about writers refrain from > > populating the stats if a special value is encountered? > > > > That fix does not preclude a more thorough solution in the future, but it > > addresses the common case quickly. > > > > For existing data files we could check the writer version ignore filters > on > > float/double. I don't know whether min/max filtering is common on > > float/double, but I suspect it's not. > > > > On Fri, Feb 16, 2018 at 8:38 AM, Tim Armstrong <tarmstr...@cloudera.com> > > wrote: > > > > > There is an extensibility mechanism with the ColumnOrder union - I > think > > > that was meant to avoid the need to add new stat fields? > > > > > > Given that the bug was in the Parquet spec, we'll need to make a spec > > > change anyway, so we could add a new ColumnOrder - > > FloatingPointTotalOrder? > > > at the same time as fixing the gap in the spec. > > > > > > It could make sense to declare that the default ordering for > > floats/doubles > > > is not NaN-aware (i.e. the reader should assume that NaN was > arbitrarily > > > ordered) and readers should either implement the required logic to > handle > > > that correctly (I had some ideas here: > > > https://issues.apache.org/jira/browse/IMPALA-6527? > > > focusedCommentId=16366106&page=com.atlassian.jira. > > > plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16366106) > > > or ignore the stats. > > > > > > On Fri, Feb 16, 2018 at 8:15 AM, Jim Apple <jbap...@cloudera.com> > wrote: > > > > > > > > We could have a similar problem > > > > > with not finding +0.0 values because a -0.0 is written to the > > max_value > > > > > field by some component that considers them the same. > > > > > > > > My hope is that the filtering would behave sanely, since -0.0 == +0.0 > > > > under the real-number-inspired ordering, which is distinguished from > > > > total Ordering, and which is also what you get when you use the > > > > default C/C++ operators <, >, <=, ==, and so on. > > > > > > > > You can distinguish between -0.0 and +0.0 without using total > ordering > > > > by taking their reciprocal: 1.0/-0.0 is -inf. There are some other > > > > ways to distinguish, I suspect, but that's the simplest one I recall > > > > at the moment. > > > > > > > > > >
