I have started organizing a project[1] in arrow-rs 's Parquet reader to try and implement this proposal.
Hopefully that can be 1 / 2 open source implementations needed. Thanks again for helping drive this along, Andrew [1] https://github.com/apache/arrow-rs/issues/8156 On Wed, Aug 13, 2025 at 5:39 AM Jan Finis <jpfi...@gmail.com> wrote: > I have now tagged > <https://github.com/apache/parquet-format/pull/514#issuecomment-3182978173 > > > the people that argued for total order in the initial PR. Let's see their > response. > > If I understand the adoption process correctly, the next hurdle to getting > this adopted is two open > source (!) implementations proving its feasibility. We already had that for > IEEE total order. If we > prefer the solution with nan counts, we'll need it there as well. I myself > work on a proprietary > implementation, so I'm counting on others here :). Be prepared though, this > will likely take months > unless the interest in this topic has risen to a point where people are > eager to jump on the implementation > right away. > > So, I guess it will take some months of soaking time before any formal vote > can be done > (given that we reach consensus that this is what we want and we find people > for the implementations). > > Cheers, > Jan > > Am Mi., 13. Aug. 2025 um 01:18 Uhr schrieb Ryan Blue <rdb...@gmail.com>: > > > Thanks, Jan. I also went through the combined proposal and it looks > mostly > > good to me. > > > > > First of all, to make it quick: Yes, the solution of having nan_counts > > *and* total order, which was brought up multiple times, does work and > > solves more cases than just either of both. > > > > Great, then we have a solution for both filtering use cases and for > moving > > ahead with total order. And thanks to Andrew for suggesting this as well > on > > the second PR. I think this also looks like this is something that Orson > is > > okay with given his comments on the latest PR. > > > > Is there anyone against the combined approach? I don't see a big downside > > for anyone. It is compatible with previous stats rules, has a NaN count, > > and allows using either type-specific order or total order. > > > > Assuming that this satisfies the big objections, I think we should wait > for > > a few days to make sure everyone has time to check out the new PR and > then > > vote to adopt it. > > > > Ryan > > > > On Mon, Aug 11, 2025 at 6:03 AM Andrew Lamb <andrewlam...@gmail.com> > > wrote: > > > > > Thank you Jan -- I read through the new combined proposal, and I > thought > > it > > > looks good and addresses the feedback so far. I left some small style > > > suggestions, but nothing that is required from my perspective > > > > > > > > > > > > On Sat, Aug 9, 2025 at 9:07 AM Jan Finis <jpfi...@gmail.com> wrote: > > > > > > > Hey Ryan, > > > > > > > > Thanks for chiming in. First of all, to make it quick: Yes, the > > solution > > > of > > > > having nan_counts *and* total order, which was brought up multiple > > times, > > > > does work and solves more cases than just either of both. > > > > > > > > I strongly prefer continuing to discuss the merits of these > approaches > > > > > rather than trying to decide with a vote. > > > > > > > > > > > > In theory, I agree that it isn't good to silence a discussion by just > > > > voting for one possible solution and technical issues should be > > > discussed. > > > > However, please note that we have been circling on this for over two > > > years > > > > now, including an extended discussion that brought up all arguments > > > > multiple times. This is in stark contrast to the > > > > speed with which you guys work on the Iceberg spec, for example. > There, > > > you > > > > also do not discuss the merits of various solutions for multiple > years. > > > You > > > > just pick one and merge it after a *reasonable* time of discussion. > > > > If you had the speed we currently have here, nothing would get done. > > > Thus, > > > > I see this as a clear case of *"the perfect is the enemy of the > good"*. > > > > Yes, we can continue looking for the perfect solution, > > > > but that will likely lead to keeping us at the status quo, which is > the > > > > worst of them all. > > > > > > > > That being said, I'm also happy to create a PR which does both total > > > order > > > > and NaN counts; after all, I just want the issue solved and all these > > > > solutions are better than the status quo. > > > > > > > > *As this was now suggest by at least three people, I guess it's worth > > > > doing, so here you go: > > https://github.com/apache/parquet-format/pull/514 > > > > <https://github.com/apache/parquet-format/pull/514>* > > > > > > > > With this, we should have PRs covering most of the solution space. > > > > (I'm refusing to create a PR with negative and positive nan_counts; > > > > nan_counts + total order has to suffice; the complexity madness has > to > > > stop > > > > somewhere) > > > > I still believe that there was an amount of people who already found > > > > nan_counts too complex and therefore wanted IEEE total order, and > these > > > > people may not like putting on extra complexity, > > > > but let's see, maybe some have also changed their opinion in the > > > meantime. > > > > > > > > > > > > *Given all this, we can also first do an informal vote where everyone > > can > > > > vote for which of the three their favorite would be.Maybe a clear > > > favorite > > > > will emerge and then we can vote on this one.* > > > > > > > > But of course, we can also take some weeks to discuss the three > > > solutions, > > > > now that we have PRs for all of them. I just hope this won't make us > > > > continue for another 2 years, or an > > > > infinite stalemate where each solution is vetoed by a PMC member. > > > > (Sorry for becoming a bit cynical here; I have just spent way too > much > > > time > > > > of my life with double statistics at this point ;) ...) > > > > > > > > > > > > Cheers, > > > > Jan > > > > > > > > Am Fr., 8. Aug. 2025 um 23:38 Uhr schrieb Ryan Blue < > rdb...@gmail.com > > >: > > > > > > > > > Regarding the process for this, I strongly prefer continuing to > > discuss > > > > the > > > > > merits of these approaches rather than trying to decide with a > vote. > > I > > > > > don't think it is a good practice to use a vote to decide on a > > > technical > > > > > direction. There are very few situations that warrant it and I > don't > > > > think > > > > > that this is one of them. While this issue has been open for a long > > > time, > > > > > that appears to be the result of it not being anyone's top priority > > > > rather > > > > > than indecision. > > > > > > > > > > For the technical merits of these approaches, I think that we can > > find > > > a > > > > > middle ground. I agree with Jan that when working with sorted > values, > > > we > > > > > need to know how NaN values were handled and that requires using a > > > > > well-defined order that includes NaN and its variations (because we > > > > should > > > > > not normalize). Using NaN count is not sufficient for ordering > rows. > > > > > > > > > > Gijs also brings up good points about how NaN values show up in > > actual > > > > > datasets: not just when used in place of null, but also as the > result > > > of > > > > > normal calculations on abnormal data, like `sqrt(-4.0)` or > > `log(-1.0)`. > > > > > Both of those present problems when mixed with valid data because > of > > > the > > > > > stats "poisoning" problem, where the range of valid data is usable > > > until > > > > a > > > > > single NaN is mixed in. > > > > > > > > > > Another issue is that NaN is error-prone because "regular" > comparison > > > is > > > > > always false: > > > > > ``` > > > > > Math.log(-1.0) >= 2 => FALSE > > > > > Math.log(-1.0) < 2 => FALSE > > > > > 2 > Math.log(-1.0) => FALSE > > > > > ``` > > > > > > > > > > As a result, Iceberg doesn't trust NaN values as either lower or > > upper > > > > > bounds because we don't want to go back to the code that produced > the > > > > value > > > > > to see what the comparison order was to determine whether NaN > values > > go > > > > > before or after others. > > > > > > > > > > Total order solves the second issue in theory, but regular > comparison > > > is > > > > > prevalent and not obvious to developers. And it also doesn't help > > when > > > > NaN > > > > > is used instead of null. So using total order is not sufficient for > > > data > > > > > skipping. > > > > > > > > > > I think the right compromise is to use `min`, `max`, and > `nan_count` > > > for > > > > > data skipping stats (where min and max cannot be NaN) and total > > > ordering > > > > > for sorting values. That satisfies the data skipping use cases and > > also > > > > > gives us an ordering of unaltered values that we can reason about. > > > > > > > > > > Does anyone think that doesn't work? > > > > > > > > > > Ryan > > > > > > > > > > On Fri, Aug 1, 2025 at 8:57 AM Gang Wu <ust...@gmail.com> wrote: > > > > > > > > > > > Thanks Jan for your endless effort on this! > > > > > > > > > > > > I'm in favor of simplicity and generalism. I think we have > already > > > > > debated > > > > > > a lot > > > > > > for `nan_count` in [1] and [2] is the reflection of those > > > discussions. > > > > > > Therefore > > > > > > I am inclined to start a vote for [2] unless there is a > > significantly > > > > > > better > > > > > > proposal. > > > > > > > > > > > > I would suggest everyone interested in this discussion to attend > > the > > > > > > scheduled > > > > > > sync on Aug 6th (detailed below) to spread the word to the > broader > > > > > > community. > > > > > > If we can get a consensus on [2], I can help start the vote and > > move > > > > > > forward. > > > > > > > > > > > > *Apache Parquet Community Sync Wednesday, August 6 · 10:00 – > > 11:00am > > > * > > > > > > *Time zone: America/Los_Angeles* > > > > > > *Google Meet joining info Video call link: > > > > > > https://meet.google.com/bhe-rvan-qjk > > > > > > <https://meet.google.com/bhe-rvan-qjk> * > > > > > > > > > > > > [1] https://github.com/apache/parquet-format/pull/196 > > > > > > [2] https://github.com/apache/parquet-format/pull/221 > > > > > > > > > > > > Best, > > > > > > Gang > > > > > > > > > > > > > > > > > > On Fri, Aug 1, 2025 at 6:16 PM Jan Finis <jpfi...@gmail.com> > > wrote: > > > > > > > > > > > > > Hi Gijs, > > > > > > > > > > > > > > Thank you for bringing up concrete points, I'm happy to discuss > > > them > > > > in > > > > > > > detail. > > > > > > > > > > > > > > NaNs are less common in the SQL world than in the DataFrame > world > > > > where > > > > > > > > NaNs were used for a long time to represent missing values. > > > > > > > > > > > > > > > > > > > > > You could transcode between NULL to NaN before reading and > > writing > > > to > > > > > > > Parquet. You basically mention yourself that NaNs were used for > > > > missing > > > > > > > values, i.e., what is commonly a NULL, which wasn't available. > > So, > > > > > > > semantically, transcoding to NULL would even be the sane thing > to > > > do. > > > > > > Yes, > > > > > > > that will cost you some cycles, but should be a rather > > lightweight > > > > > > > operation in comparison to most other operations, so I would > > argue > > > > that > > > > > > it > > > > > > > won't totally ruin your performance. Similarly, why should > > Parquet > > > > play > > > > > > > along with a "hack" that was done in other frameworks due to > > > > > shortcomings > > > > > > > of those frameworks? So from a philosophical point of view, I > > think > > > > > > > supporting NaNs better is the wrong thing to do. Rather, we > > should > > > > be a > > > > > > > forcing function to align others to better behavior, so > appling a > > > bit > > > > > of > > > > > > > force might in the long run make people use NULLs also in > > > DataFrames. > > > > > > > > > > > > > > Of course, your argument also goes into the direction of > > > pragmatism: > > > > > If a > > > > > > > large part of the data science world uses NaNs to encode > missing > > > > > values, > > > > > > > then maybe Parquet should accept this de-facto standard rather > > than > > > > > > > fighting it. That is indeed a valid point. The weight of it is > > > > > debatable > > > > > > > and my personal conclusion is that it's still not worth it, as > > you > > > > can > > > > > > > transcode between NULLs and NaNs, but I do agree with its > > validity. > > > > > > > > > > > > > > > > > > > > > Since the proposal phrases it as a goal to work "regardless of > > how > > > > they > > > > > > > > order NaN w.r.t. other values" this statement feels > > out-of-place > > > to > > > > > me. > > > > > > > > Most hardware and most people don't care about total ordering > > and > > > > > > needing > > > > > > > > to take it into account while filtering using statistics > seems > > > like > > > > > > > > preferring the special case instead of the common case. > Almost > > > > noone > > > > > > > > filters for specific NaN value bit-patterns. SQL engines that > > > don't > > > > > > have > > > > > > > > IEEE total ordering as their default ordering for floats will > > > also > > > > > need > > > > > > > to > > > > > > > > do more special handling for this. > > > > > > > > > > > > > > > > > > > > > I disagree with the conclusion this statement draws. The > current > > > > > > behavior, > > > > > > > and nan_counts without total ordering, pose a real problem > here, > > > even > > > > > for > > > > > > > engines that don't care about bit patterns. I do agree that > most > > > > > database > > > > > > > engines, including the one I'm working on, do not care about > bit > > > > > patterns > > > > > > > and/or sign bits. However, how can our database engine know > > whether > > > > the > > > > > > > writer of a Parquet file saw it the same way? It can't. > > Therefore, > > > it > > > > > > > cannot know whether a writer, for example, ordered NaNs before > or > > > > after > > > > > > all > > > > > > > other numbers, or maybe ordered them by sign bit. So, if our > > > database > > > > > > > engine now sees a float column in sorting columns, it cannot > > apply > > > > any > > > > > > > optimization without a lot of special casing, as it doesn't > know > > > > > whether > > > > > > > NaNs will be before all other values, after all other values, > or > > > > maybe > > > > > > > both, depending on sign bit. It could apply contrived logic > that > > > > tries > > > > > to > > > > > > > infer where NaNs were placed from the NaN counts of the first > and > > > > last > > > > > > > page, but doing so will be a lot of ugly code that also feels > to > > be > > > > in > > > > > > the > > > > > > > wrong place. I.e., I don't want to need to load pages or the > page > > > > > index, > > > > > > > just to reason about a sort order. > > > > > > > > > > > > > > SQL engines that don't have > > > > > > > > IEEE total ordering as their default ordering for floats will > > > also > > > > > need > > > > > > > to > > > > > > > > do more special handling for this. > > > > > > > > > > > > > > > > > > > > > This code, which I would indeed need to write for our engine, > is > > > > > > comparably > > > > > > > trivial. Simply choose the largest possible bit pattern as > > > comparison > > > > > for > > > > > > > upper bounds filtering for NaN, and the smallest possible bit > > > pattern > > > > > for > > > > > > > lower bounds. It's not more than a few lines of code that check > > > > > whether a > > > > > > > filter is NaN and then replace its value with the > highest/lowest > > > NaN > > > > > bit > > > > > > > pattern. It is similarly trivial to the special casing I need > to > > do > > > > > with > > > > > > > nan_counts, and it is way more trivial than the extra code I > > would > > > > need > > > > > > to > > > > > > > write for sorting columns, as depicted above. > > > > > > > > > > > > > > From a Polars perspective, having a `nan_count` and defining > what > > > > > > > > happens to the `min` and `max` statistics when a page > contains > > > only > > > > > > NaNs > > > > > > > is > > > > > > > > enough to allow for all predicate filtering. I think, but > > correct > > > > me > > > > > > if I > > > > > > > > am wrong, this is also enough for all SQL engines that don't > > use > > > > > total > > > > > > > > ordering. > > > > > > > > > > > > > > > > > > > > > It's not fully enough, as depicted above. Sorting columns would > > > still > > > > > not > > > > > > > work properly. > > > > > > > > > > > > > > As for ways forward, I propose merging the `nan_count` and > `sort > > > > > > ordering` > > > > > > > > proposals into one to make one proposal > > > > > > > > > > > > > > > > > > > > > Note that the initial reason for proposing IEEE total order was > > > that > > > > > > people > > > > > > > in the discussion threads found nan_counts to be too complex > and > > > too > > > > > much > > > > > > > of an undeserving special case (re-read the discussion in the > > > initial > > > > > PR > > > > > > > <https://github.com/apache/parquet-format/pull/196> to see the > > > > > > > rationales). > > > > > > > So merging both together would go totally against the spirit of > > why > > > > > IEEE > > > > > > > total order was proposed. While it has further upsides, the > main > > > > reason > > > > > > was > > > > > > > indeed to *not have* nan_counts. If now the proposal would even > > go > > > to > > > > > > > positive and negative nan counts (i.e., even more complexity), > > this > > > > > would > > > > > > > go 180 degrees into the opposite direction of why people wanted > > > total > > > > > > order > > > > > > > in the first place. > > > > > > > > > > > > > > Cheers, > > > > > > > Jan > > > > > > > > > > > > > > Am Do., 31. Juli 2025 um 23:23 Uhr schrieb Gijs Burghoorn > > > > > > > <g...@polars.tech.invalid>: > > > > > > > > > > > > > > > Hello Jan and others, > > > > > > > > > > > > > > > > First, let me preface by saying I am quite new here. So I > > > apologize > > > > > if > > > > > > > > there is some other better way to bring up these concerns. I > > > > > understand > > > > > > > it > > > > > > > > is very annoying to come in at the 11th hour and start > bringing > > > up > > > > a > > > > > > > bunch > > > > > > > > of concerns, but I would also like this to be done right. A > > > > colleague > > > > > > of > > > > > > > > mine brought up some concerns and alternative approaches in > the > > > > > GitHub > > > > > > > > thread; I will file some of the concerns here as a response. > > > > > > > > > > > > > > > > > Treating NaNs so specially is giving them attention they > > don't > > > > > > deserve. > > > > > > > > Most data sets do not contain NaNs. If a use case really > > requires > > > > > them > > > > > > > and > > > > > > > > needs filtering to ignore them, they can store NULL instead, > or > > > > > encode > > > > > > > them > > > > > > > > differently. I would prefer the average case over the special > > > case > > > > > > here. > > > > > > > > > > > > > > > > NaNs are less common in the SQL world than in the DataFrame > > world > > > > > where > > > > > > > > NaNs were used for a long time to represent missing values. > > They > > > > > still > > > > > > > > exist with different canonical representations and different > > sign > > > > > > bits. I > > > > > > > > agree it might not be correct semantically, but sadly that is > > the > > > > > world > > > > > > > we > > > > > > > > deal with. NumPy and Numba do not have missing data > > > functionality, > > > > > > people > > > > > > > > use NaNs there, and people definitely use that in their > > > analytical > > > > > > > > dataflows. Another point that was brought up in the GH > > discussion > > > > was > > > > > > > "what > > > > > > > > about infinity? You could argue that having infinity in > > > statistics > > > > is > > > > > > > > similarly unuseful as it's too wide of a bound". I would > argue > > > that > > > > > > > > infinity is very different as there is no discussion on what > > the > > > > > > ordering > > > > > > > > or pattern of infinity is. Everyone agrees that `min(1.0, > inf, > > > > -inf) > > > > > == > > > > > > > > -inf` and each infinity only has a single bit pattern. > > > > > > > > > > > > > > > > > It gives a defined order to every bit pattern and thus > > yields a > > > > > total > > > > > > > > order, mathematically speaking, which has value by itself. > With > > > NaN > > > > > > > counts, > > > > > > > > it was still undefined how different bit patterns of NaNs > were > > > > > supposed > > > > > > > to > > > > > > > > be ordered, whether NaN was allowed to have a sign bit, etc., > > > > risking > > > > > > > that > > > > > > > > different engines could come to different results while > > filtering > > > > or > > > > > > > > sorting values within a file. > > > > > > > > > > > > > > > > Since the proposal phrases it as a goal to work "regardless > of > > > how > > > > > they > > > > > > > > order NaN w.r.t. other values" this statement feels > > out-of-place > > > to > > > > > me. > > > > > > > > Most hardware and most people don't care about total ordering > > and > > > > > > needing > > > > > > > > to take it into account while filtering using statistics > seems > > > like > > > > > > > > preferring the special case instead of the common case. > Almost > > > > noone > > > > > > > > filters for specific NaN value bit-patterns. SQL engines that > > > don't > > > > > > have > > > > > > > > IEEE total ordering as their default ordering for floats will > > > also > > > > > need > > > > > > > to > > > > > > > > do more special handling for this. > > > > > > > > > > > > > > > > I also agree with my colleague that doing an approach that is > > 50% > > > > of > > > > > > the > > > > > > > > way there will make the barrier to improving it to what it > > > actually > > > > > > > should > > > > > > > > be later on much higher. > > > > > > > > > > > > > > > > As for ways forward, I propose merging the `nan_count` and > > `sort > > > > > > > ordering` > > > > > > > > proposals into one to make one proposal, as they are linked > > > > together, > > > > > > and > > > > > > > > moving forward with one without knowing what will happen to > the > > > > other > > > > > > > seems > > > > > > > > unwise. From a Polars perspective, having a `nan_count` and > > > > defining > > > > > > what > > > > > > > > happens to the `min` and `max` statistics when a page > contains > > > only > > > > > > NaNs > > > > > > > is > > > > > > > > enough to allow for all predicate filtering. I think, but > > correct > > > > me > > > > > > if I > > > > > > > > am wrong, this is also enough for all SQL engines that don't > > use > > > > > total > > > > > > > > ordering. But if you want to be impartial to the engine's > > > > > > floating-point > > > > > > > > ordering and allow engines with total ordering to do > inequality > > > > > filters > > > > > > > > when `nan_count > 0` you would need a `positive_nan_count` > and > > a > > > > > > > > `negative_nan_count`. I understand the downside with Thrift > > > > > complexity, > > > > > > > but > > > > > > > > introducing another sort order is also adding complexity just > > in > > > a > > > > > > > > different place. > > > > > > > > > > > > > > > > I would really like to see this move forward, so I hope these > > > > > concerns > > > > > > > help > > > > > > > > move it forward towards a solution that works for everyone. > > > > > > > > > > > > > > > > Kind regards, > > > > > > > > Gijs > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 31, 2025 at 7:46 PM Andrew Lamb < > > > > andrewlam...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I would also be in favor of starting a vote > > > > > > > > > > > > > > > > > > On Thu, Jul 31, 2025 at 11:23 AM Jan Finis < > > jpfi...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > As the author of both the IEEE754 total order > > > > > > > > > > <https://github.com/apache/parquet-format/pull/221> PR > and > > > the > > > > > > > earlier > > > > > > > > > PR > > > > > > > > > > that basically proposed `nan_count` > > > > > > > > > > <https://github.com/apache/parquet-format/pull/196>, my > > > > current > > > > > > vote > > > > > > > > > would > > > > > > > > > > be for IEEE754 total order. > > > > > > > > > > Consequently, I would like to request a formal vote for > the > > > PR > > > > > > > > > introducing > > > > > > > > > > IEEE754 total order ( > > > > > > > https://github.com/apache/parquet-format/pull/221 > > > > > > > > ), > > > > > > > > > > if > > > > > > > > > > that is possible. > > > > > > > > > > > > > > > > > > > > My Rationales: > > > > > > > > > > > > > > > > > > > > - It's conceptually simpler. It's easier to explain. > > It's > > > > > based > > > > > > on > > > > > > > > an > > > > > > > > > > IEEE-standardized order predicate. > > > > > > > > > > - There are already multiple implementations showing > > > > > > feasibility. > > > > > > > > This > > > > > > > > > > will likely make the adoption quicker. > > > > > > > > > > - It gives a defined order to every bit pattern and > thus > > > > > yields > > > > > > a > > > > > > > > > total > > > > > > > > > > order, mathematically speaking, which has value by > > itself. > > > > > With > > > > > > > NaN > > > > > > > > > > counts, > > > > > > > > > > it was still undefined how different bit patterns of > > NaNs > > > > were > > > > > > > > > supposed > > > > > > > > > > to > > > > > > > > > > be ordered, whether NaN was allowed to have a sign > bit, > > > > etc., > > > > > > > > risking > > > > > > > > > > that > > > > > > > > > > different engines could come to different results > while > > > > > > filtering > > > > > > > or > > > > > > > > > > sorting values within a file. > > > > > > > > > > - It also solves sort order completely. With > nan_counts > > > > only, > > > > > it > > > > > > > is > > > > > > > > > > still undefined whether nans should be sorted before > or > > > > after > > > > > > all > > > > > > > > > values > > > > > > > > > > (or both, depending on sign bit), so any file > including > > > NaNs > > > > > > could > > > > > > > > not > > > > > > > > > > really leverage sort order without being ambiguous. > > > > > > > > > > - It's less complex in thrift. Having fields that only > > > apply > > > > > to > > > > > > a > > > > > > > > > > handful of data types is somehow weird. If every type > > did > > > > > this, > > > > > > we > > > > > > > > > would > > > > > > > > > > have a plethora of non-generic fields in thrift. > > > > > > > > > > - Treating NaNs so specially is giving them attention > > they > > > > > don't > > > > > > > > > > deserve. Most data sets do not contain NaNs. If a use > > case > > > > > > really > > > > > > > > > > requires > > > > > > > > > > them and needs filtering to ignore them, they can > store > > > NULL > > > > > > > > instead, > > > > > > > > > > or encode them differently. I would prefer the average > > > case > > > > > over > > > > > > > the > > > > > > > > > > special case here. > > > > > > > > > > - The majority of the people discussing this so far > seem > > > to > > > > > > favor > > > > > > > > > total > > > > > > > > > > order. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Jan > > > > > > > > > > > > > > > > > > > > Am Sa., 26. Juli 2025 um 17:38 Uhr schrieb Gang Wu < > > > > > > ust...@gmail.com > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > As this discussion has been open for more than two > years, > > > I’d > > > > > > like > > > > > > > to > > > > > > > > > > bump > > > > > > > > > > > up > > > > > > > > > > > this thread again to update the progress and collect > > > > feedback. > > > > > > > > > > > > > > > > > > > > > > *Background* > > > > > > > > > > > • Today Parquet’s min/max stats and page index omit > NaNs > > > > > > entirely. > > > > > > > > > > > • Engines can’t safely prune floating values because > they > > > > know > > > > > > > > nothing > > > > > > > > > on > > > > > > > > > > > NaNs. > > > > > > > > > > > • Column index is disabled if any page contains only > > NaNs. > > > > > > > > > > > > > > > > > > > > > > There are two active proposals as below: > > > > > > > > > > > > > > > > > > > > > > *Proposal A - IEEE754TotalOrder* (from the PR [1]) > > > > > > > > > > > • Define a new ColumnOrder to include +0, –0 and all > NaN > > > > > > > > bit‐patterns. > > > > > > > > > > > • Stats and column index store NaNs if they appear. > > > > > > > > > > > • Three PoC impls are ready: arrow-rs [2], duckdb [3] > and > > > > > > > > parquet-java > > > > > > > > > > [4]. > > > > > > > > > > > • For more context of this approach, please refer to > > > > discussion > > > > > > in > > > > > > > > [5]. > > > > > > > > > > > > > > > > > > > > > > *Proposal B - add nan_count* (from a comment [6] to > [1]) > > > > > > > > > > > • Add `nan_count` to stats and a `nan_counts` list to > > > column > > > > > > index. > > > > > > > > > > > • For all‐NaNs cases, write NaN to min/max and use > > > nan_count > > > > to > > > > > > > > > > > distinguish. > > > > > > > > > > > > > > > > > > > > > > Both solutions have pros and cons but are way better > than > > > the > > > > > > > status > > > > > > > > > quo > > > > > > > > > > > today. > > > > > > > > > > > Please share your thoughts on the two proposals above, > or > > > > maybe > > > > > > > come > > > > > > > > up > > > > > > > > > > > with > > > > > > > > > > > better alternatives. We need consensus on one proposal > > and > > > > move > > > > > > > > > forward. > > > > > > > > > > > > > > > > > > > > > > [1] https://github.com/apache/parquet-format/pull/221 > > > > > > > > > > > [2] https://github.com/apache/arrow-rs/pull/7408 > > > > > > > > > > > [3] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/duckdb/duckdb/compare/main...Mytherin:duckdb:ieeeorder > > > > > > > > > > > [4] https://github.com/apache/parquet-java/pull/3191 > > > > > > > > > > > [5] https://github.com/apache/parquet-format/pull/196 > > > > > > > > > > > [6] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/parquet-format/pull/221#issuecomment-2931376077 > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Gang > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 28, 2023 at 4:22 PM Jan Finis < > > > jpfi...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Dear contributors, > > > > > > > > > > > > > > > > > > > > > > > > My PR has now gathered comments for a week and the > gist > > > of > > > > > all > > > > > > > open > > > > > > > > > > > issues > > > > > > > > > > > > is the question of how to encode pages/column chunks > > that > > > > > > contain > > > > > > > > > only > > > > > > > > > > > > NaNs. There are different suggestions and I don't see > > one > > > > > > common > > > > > > > > > > favorite > > > > > > > > > > > > yet. > > > > > > > > > > > > > > > > > > > > > > > > I have outlined three alternatives of how we can > handle > > > > these > > > > > > > and I > > > > > > > > > > want > > > > > > > > > > > us > > > > > > > > > > > > to reach a conclusion here, so I can update my PR > > > > accordingly > > > > > > and > > > > > > > > > move > > > > > > > > > > on > > > > > > > > > > > > with it. As this is my first contribution to > parquet, I > > > > don't > > > > > > > know > > > > > > > > > the > > > > > > > > > > > > decision processes here. Do we vote? Is there a > single > > or > > > > > group > > > > > > > of > > > > > > > > > > > decision > > > > > > > > > > > > makers? *Please let me know how to come to a > conclusion > > > > here; > > > > > > > what > > > > > > > > > are > > > > > > > > > > > the > > > > > > > > > > > > next steps?* > > > > > > > > > > > > > > > > > > > > > > > > For reference, here are the three alternatives I > > pointed > > > > out. > > > > > > You > > > > > > > > can > > > > > > > > > > > find > > > > > > > > > > > > detailed description of their PROs and CONs in my > > > comment: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/parquet-format/pull/196#issuecomment-1486416762 > > > > > > > > > > > > > > > > > > > > > > > > 1. My initial proposal, i.e., encoding only-NaN pages > > by > > > > > > > > min=max=NaN. > > > > > > > > > > > > 2. Adding `num_values` to the ColumnIndex, to make it > > > > > symmetric > > > > > > > > with > > > > > > > > > > > > Statistics in pages & `ColumnMetaData` and to enable > > the > > > > > > > > computation > > > > > > > > > > > > `num_values - null_count - nan_count == 0` > > > > > > > > > > > > 3. Adding a `nan_pages` bool list to the column > index, > > > > which > > > > > > > > > indicates > > > > > > > > > > > > whether a page contains only NaNs > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > > > > Jan Finis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
