I feel like we're reading into this too much to justify changing the definition. I was involved in the project when this was introduced (over 13 years ago), but I have no recollection of discussing the kinds of concerns we're raising now. Issues like character uniqueness, equivalence, estimates were very unlikely to have been considered at the time. However, the lack of a clear statement of fact that the value is expected to be exact, is no justification to assume that an estimate is a reasonable replacement.
There are a lot of issues with using distinct_count, for example, it's not combinable across row groups or files, many implementations don't produce it and likely fewer consume it. However, none of that is justification in light of what it currently states and is expected to contain. As I see it, there are only two paths forward as I previously stated: 1) introduce a new field, or 2) change the definition in conjunction with a forward compatibility change (as discussed in the versioning topic). -Dan On Tue, Nov 10, 2026 at 4:06 PM Adrian Garcia Badaracco via dev < [email protected]> wrote: > Hi Dan, > > Maybe I’m just missing context here. Were you part of introducing the > field? Could you point me at info on where this field was ever understood > to be exact? That might help me see the point of view that there was any > expectation of this field being exact. > > Adrian > > > On Nov 10, 2026, at 12:30 PM, Daniel Weeks via dev < > [email protected]> wrote: > > > > Hey Adrian, > > > > I'm ok with a clarification, but I don't think updating the definition to > > be an estimate is inline with that. > > > > At this point, it feels like we're trying to justify changing the > > definition (it was never clearly specified, it was never defined as > > accurate, etc.) as opposed to looking at the intent behind the > > introduction. It's very likely that we didn't understand the nuance > > or complications at the time, but I don't think anyone understood it to > be > > an estimate. > > > > If we want to clarify it may not be accurate and could cause incorrect > > results if relied on under certain cases, that would fall within a > > compatible change. If we want to make an estimate, I believe it would > have > > to be a new field. > > > > It would be "easier" to just redefine it, but that's not compatible with > > the guarantees we need to provide. That leaves us with creating a new > > field if we want a value understood to be an estimate. > > > > The only reasonable alternative would be to wait until we agree on the > next > > major version at which point we could redefine the value (since a version > > bump allows for backward incompatible changes like the proposed footer). > > > > Dan > > > > > > > > > > On Mon, Nov 9, 2026 at 7:20 AM Adrian Garcia Badaracco < > [email protected]> > > wrote: > > > >> Hi Dan, > >> > >> I understand the concern about redefining a field. I see this as > defining > >> the field for the first time. The field was never defined as accurate, > >> accuracy was simply never specified. No reader expects it to be > accurate, > >> and no writer writes completely accurate values. > >> > >> There is precedent for a similar change for zone map min/max stats: in > >> 2023, PARQUET-2352 (#216 < > >> https://github.com/apache/parquet-format/pull/216>) changed `min_value` > >> and `max_value` from "Min and max values" to "Lower and upper bound > values". > >> > >> If you still prefer not to change the text of `distinct_count`, I am > fine > >> with the alternative: deprecate `distinct_count` and add a new field. I > >> would propose that the new field be `estimated_distinct_count`, defined > as > >> “An estimate of the number of distinct non-null values in the page or > >> column chunk. The estimate can be higher or lower than the actual count > by > >> any amount.". This field would be useful for join ordering estimation, > >> which is the use case multiple systems are using the existing field > for. It > >> would be easy and performant to compute and use for both readers and > >> writers, and the harm when it is grossly wrong is limited. > >> > >> The main alternative I can think of is a sketch. A sketch can be merged > >> across row groups and files, which helps for single-column joins. But > it is > >> larger, more complex to define, and needs agreement on a sketch format. > It > >> also does not help much for joins on multiple columns and other more > >> complex cases. So I think starting with a simple per-column per-chunk > >> estimated distinct count integer and adding a sketch later if there are > >> good use cases for it would be the better path forward. > >> > >> Do you see us agreeing on one of these two paths (update the definition > to > >> be an estimate or deprecate and add a new field)? > >> > >> Thanks, > >> > >> Adrian > >> > >> On 2026/10/19 14:33:14 Daniel Weeks wrote: > >>> Adrian, I think this issue goes back to Andrew's initial response, > which > >> I > >>> agree with: > >>> > >>> However, I think redefining the existing distinct_count statistic > >>>> would have all the forward compatibility problems we have been > debating > >>> > >>> > >>> Looking at existing usage and leveraging that to argue changing the > >> meaning > >>> of a field undermines the guarantees we try to provide as a format. > >>> > >>> I can understand clarifying the definition, but this feels like we're > >>> trying to convince ourselves it's ok to redefine the field. > >>> > >>> There are a few scenarios we've discussed where it's reasonable that > >> value > >>> could be ambiguous, but I think generally the expectation should be > that > >> it > >>> is accurate. > >>> > >>> If we think there's a better use case for carrying estimated values, > >> maybe > >>> that's good justification for adding a new field (though with the > limited > >>> usage you identified, maybe it's not worth the effort). > >>> > >>> -Dan > >>> > >>> On Thu, Oct 15, 2026 at 1:43 PM Adrian Garcia Badaracco < > >> [email protected]> > >>> wrote: > >>> > >>>> To answer Micah's question, I looked at what current implementations > do > >>>> with distinct_count, and how engines use NDVs from other sources. > >>>> I am familiar with DataFusion/arrow-rs, but for other systems (DuckDB, > >>>> Polars, Arrow C++, parquet-java, Spark, Trino, Iceberg, Hive and > >> Impala) I > >>>> used an LLM to search their current implementation and did not verify > >> the > >>>> findings by hand. > >>>> Please correct me if I got any of them wrong. > >>>> > >>>> Who writes distinct_count: > >>>> - arrow-rs: opt-in, off by default. It counts a set of value hashes. > >>>> - DuckDB: only for dictionary-encoded chunks (it writes the dictionary > >>>> size). > >>>> - Polars: only for boolean columns. > >>>> - parquet-java, Spark, Trino, Iceberg, Hive and Impala: never. > >>>> > >>>> Who reads it: > >>>> - DataFusion and Polars, and only as a planner estimate. > >>>> - No reader I found uses it for correctness. > >>>> > >>>> Current values are already not exact: > >>>> - Signed zero: arrow-rs and DuckDB both count -0.0 and +0.0 as two > >> values. > >>>> - NaN: arrow-rs compares raw bytes, so NaNs with different bit > patterns > >>>> count as different values. > >>>> - Hash collisions: arrow-rs counts xxHash64 hashes, not values, so > >>>> collisions can make the count too low. > >>>> - Dictionary arrays: arrow-rs counts dictionary keys, not values. > Arrow > >>>> doesn't require dictionary values to be unique or used, so the count > >> can be > >>>> too high. > >>>> So current writers can be wrong in both directions, and they don't > >> share a > >>>> definition of equality. > >>>> > >>>> Engines with their own NDVs use estimates: > >>>> - Spark: HLL++, 5% default error [1] > >>>> - Trino: approx_distinct (HLL, about 2.3% error) for Hive tables [2], > >> and > >>>> theta sketches for Iceberg tables > >>>> - Iceberg: Puffin theta sketches [3] > >>>> - Hive: HLL by default [4] > >>>> All of them use NDV only for cost-based planning: filter selectivity, > >> join > >>>> cardinality, and through those, join order and broadcast choice. > >>>> > >>>> In conclsion: > >>>> - Readers don't depend on distinct_count being exact. > >>>> - Engines that compute their own NDVs choose estimates on purpose. > >>>> - So I think documenting the field as an estimate matches how it is > >> used > >>>> today and covers most of the potential future use cases. > >>>> > >>>> [1] > >>>> > >> > https://github.com/apache/spark/blob/41f1694f7ba7cc3dd247d00dc42d13f5a8b1eaf7/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala#L399 > >>>> [2] > >>>> > >> > https://github.com/apache/trino/blob/cf8992f543d0671b2579fded1d17ae3ee862d543/core/trino-main/src/main/java/io/trino/operator/aggregation/DefaultApproximateCountDistinctAggregation.java#L41 > >>>> [3] > >>>> > >> > https://github.com/apache/iceberg/blob/acee558a84e8dd065bf49fa0af709a5346ede077/format/puffin-spec.md#L114-L123 > >>>> [4] > >>>> > >> > https://github.com/apache/hive/blob/da789295710cd04b7a76185c6551ec7ef5ab6f1f/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L2798-L2799 > >>>> > >>>> On 2026/10/15 20:08:22 Adrian Garcia Badaracco wrote: > >>>>> I sense the pushback is against saying that writers may use arbitrary > >>>> estimates. But unless we agree on how much error is acceptable, or on > >> how > >>>> equality is defined, we are in effect allowing an unbounded estimate > >> anyway. > >>>>> “Approximate” with no bound is the same as “arbitrary”. > >>>>> > >>>>> So I think we should pick one of these two options: > >>>>> > >>>>> 1. Define the allowed error (or the counting rules) precisely now. > >>>>> 2. Say explicitly that the value is an estimate with no guaranteed > >> bound. > >>>>> > >>>>> I propose option 2 for now. We can come back and add tighter > >> guarantees > >>>> when there are concrete use cases that need them, for example a reader > >> that > >>>> needs to know the error stays within a specific bound, or that the > >> writer > >>>> followed certain rules to compute the distinct count. > >>>>> Adding a guarantee later is backward compatible because existing > >> readers > >>>> already treat the value as a hint. Removing one would not be. > >>>>> > >>>>> On 2026/10/07 19:52:38 Daniel Weeks wrote: > >>>>>> Andrew, I agree with you that the proposed wording is probably not > >> the > >>>> best. > >>>>>> > >>>>>> I don't think the intent was to replace it with arbitrary estimates > >>>> (though > >>>>>> the current wording indicates that). > >>>>>> > >>>>>> I offered up some alternative language on the PR. > >>>>>> > >>>>>> -Dan > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Wed, Oct 7, 2026 at 12:27 PM Andrew Lamb <[email protected]> > >>>> wrote: > >>>>>> > >>>>>>> IN my mind, the fact that "distinct" is not precisely defined for > >>>> all types > >>>>>>> means that changing "count of distinct values occurring" to > >>>> "Estimated > >>>>>>> count of distinct values occurring." is a clarification > >>>>>>> > >>>>>>> It seems like a significant change in meaning (explicitly > >> allowing > >>>>>>> estimates) to me <shrug> > >>>>>>> > >>>>>>> Andrew > >>>>>>> > >>>>>>> On Wed, Oct 7, 2026 at 2:24 PM Daniel Weeks <[email protected]> > >>>> wrote: > >>>>>>> > >>>>>>>> I think the issue is less about whether this is changing > >> something > >>>> for > >>>>>>>> older readers/writers or providing a more accurate description. > >>>>>>>> > >>>>>>>> Older readers would already "misinterpret" written values > >> because > >>>>>>> there's a > >>>>>>>> difference between logical equivalence and physical > >> equivalence. > >>>> This > >>>>>>>> difference is left up to interpretation by both the reader and > >>>> writer. > >>>>>>>> > >>>>>>>> An example would be -0 vs +0 for floating point types where > >> some > >>>> systems > >>>>>>>> treat them equally where others may distinguish them. > >> Similarly > >>>> NaN may > >>>>>>> or > >>>>>>>> may not be included as a value count. > >>>>>>>> > >>>>>>>> I think it's wrong to assume that current readers/writers are > >>>> capturing > >>>>>>>> this "correctly" when there is no "correct" or agreedupon > >>>> representation. > >>>>>>>> > >>>>>>>> -Dan > >>>>>>>> > >>>>>>>> On Wed, Oct 7, 2026 at 11:23 AM Russell Spitzer < > >>>>>>> [email protected] > >>>>>>>>> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> My understanding is that the exactness of NDV is not > >>>> standardized at > >>>>>>> all > >>>>>>>>> for many types, which makes this value very difficult to > >> define > >>>> or > >>>>>>>>> standardize as is. The main callouts during the meeting were > >>>> strings > >>>>>>> with > >>>>>>>>> collations (foo == Foo) floating point (NAN == NAN, -0.0 = > >> 0.0 > >>>> etc...) > >>>>>>> So > >>>>>>>>> since we don't want to spend time addressing these issues, as > >>>> most use > >>>>>>>>> cases only require approximate statistics, the consensus was > >> to > >>>> correct > >>>>>>>> the > >>>>>>>>> language now rather than spend time establishing something > >> new. > >>>> Just a > >>>>>>>>> reminder for everyone, please make sure to add notes when > >>>> referring to > >>>>>>>>> discussions that occurred off list. I know the full video and > >>>> trascript > >>>>>>>>> will eventually be available but it's helpful to keep things > >>>> straight > >>>>>>> :) > >>>>>>>>> From the notes doc - > >>>>>>>>> > >>>>>>>>> - > >>>>>>>>> > >>>>>>>>> exactness of NDV stats: > >>>>>>>>> - > >>>>>>>>> > >>>>>>>>> Wanted to use in datafusion > >>>>>>>>> - > >>>>>>>>> > >>>>>>>>> Question of exactitude. > >>>>>>>>> - > >>>>>>>>> > >>>>>>>>> Collations makes it difficult because two distinct > >> values > >>>> might > >>>>>>> be > >>>>>>>>> considered equal. > >>>>>>>>> - > >>>>>>>>> > >>>>>>>>> Semantics are under defined. > >>>>>>>>> - > >>>>>>>>> > >>>>>>>>> Do we want to formalize for floating point numbers: > >> -0, > >>>> +0, > >>>>>>>> NaNs, > >>>>>>>>> etc? > >>>>>>>>> - > >>>>>>>>> > >>>>>>>>> Adrian: > >>>>>>>>> - > >>>>>>>>> > >>>>>>>>> Define right now as exact. > >>>>>>>>> - > >>>>>>>>> > >>>>>>>>> Most use cases do not require exact. > >>>>>>>>> - > >>>>>>>>> > >>>>>>>>> Statistical approximations of counts? > >>>>>>>>> - > >>>>>>>>> > >>>>>>>>> Kevin: > >>>>>>>>> - > >>>>>>>>> > >>>>>>>>> do we need to update the doc as well? > >>>>>>>>> - > >>>>>>>>> > >>>>>>>>> Yes! > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Wed, Oct 7, 2026 at 1:08 PM Micah Kornfield < > >>>> [email protected]> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hi Adrian, > >>>>>>>>>> Is there more context from the call, on why Andrew's > >> assessment > >>>>>>> didn't > >>>>>>>>> make > >>>>>>>>>> sense? Are there reference implementations providing > >>>> approximate > >>>>>>>> answers > >>>>>>>>>> today? > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Micah > >>>>>>>>>> > >>>>>>>>>> On Wed, Oct 7, 2026 at 10:28 AM Adrian Garcia Badaracco < > >>>>>>>>>> [email protected]> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Per discussion on the sync call today I opened a format > >> PR: > >>>>>>>>>>> https://github.com/apache/parquet-format/pull/628 > >>>>>>>>>>> > >>>>>>>>>>> On 2026/09/30 21:26:04 Andrew Lamb wrote: > >>>>>>>>>>>> I think optional approximate distinct (or other > >> sketches > >>>> of the > >>>>>>>> data > >>>>>>>>>> like > >>>>>>>>>>>> histograms) would help various database planners make > >>>> better > >>>>>>> plans, > >>>>>>>>> so > >>>>>>>>>>> they > >>>>>>>>>>>> are valuable to explore. > >>>>>>>>>>>> > >>>>>>>>>>>> However, I think redefining the existing distinct_count > >>>> statistic > >>>>>>>>>>>> would have all the forward compatibility problems we > >> have > >>>> been > >>>>>>>>> debating > >>>>>>>>>>> -- > >>>>>>>>>>>> namely we have no mechanism to prevent older readers > >>>>>>>> misinterpreting > >>>>>>>>> a > >>>>>>>>>>>> field whose semantic meaning was redefined. > >>>>>>>>>>>> > >>>>>>>>>>>> So maybe we could investigate adding new standardized > >>>> metadata > >>>>>>>>> instead > >>>>>>>>>> of > >>>>>>>>>>>> redefining the meaning of an old one > >>>>>>>>>>>> > >>>>>>>>>>>> Andrew > >>>>>>>>>>>> > >>>>>>>>>>>> On Tue, Sep 22, 2026 at 12:30 PM Adrian Garcia > >> Badaracco > >>>> via dev > >>>>>>> < > >>>>>>>>>>>> [email protected]> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> The specification does not define what > >>>>>>>> `Statistics.distinct_count` > >>>>>>>>>>>>> contains. Writers do not know if they can write an > >>>> estimate. > >>>>>>>>> Readers > >>>>>>>>>>> do not > >>>>>>>>>>>>> know if they can use the value as an exact count. > >>>>>>>>>>>>> > >>>>>>>>>>>>> The field has this definition in `parquet.thrift`: > >>>>>>>>>>>>> > >>>>>>>>>>>>> ```thrift > >>>>>>>>>>>>> /** count of distinct values occurring */ > >>>>>>>>>>>>> 4: optional i64 distinct_count; > >>>>>>>>>>>>> ``` > >>>>>>>>>>>>> > >>>>>>>>>>>>> The text does not answer these questions: > >>>>>>>>>>>>> > >>>>>>>>>>>>> 1. Must the value be exact? Or can it be an > >> estimate, for > >>>>>>> example > >>>>>>>>>> from > >>>>>>>>>>> a > >>>>>>>>>>>>> HyperLogLog sketch? > >>>>>>>>>>>>> > >>>>>>>>>>>>> If we choose to make the value inexact, this opens > >> up the > >>>>>>>> question: > >>>>>>>>>>>>> > >>>>>>>>>>>>> 1. If the value is not exact, what does it > >> guarantee? Is > >>>> it an > >>>>>>>>> upper > >>>>>>>>>>>>> bound, a lower bound, or an estimate with no bound? > >>>>>>>>>>>>> > >>>>>>>>>>>>> If we choose to make the value exact, this opens up > >> other > >>>>>>>>> questions: > >>>>>>>>>>>>> > >>>>>>>>>>>>> 1. How do nulls factor into the count? > >>>>>>>>>>>>> 2. How do writers compare values? For example, are > >>>> `-0.0` and > >>>>>>>>> `+0.0` > >>>>>>>>>>> one > >>>>>>>>>>>>> value or two? Are two NaN values with different bit > >>>> patterns > >>>>>>> one > >>>>>>>>>> value > >>>>>>>>>>> or > >>>>>>>>>>>>> two? Is the comparison on the physical bytes or on > >> the > >>>> logical > >>>>>>>>> type? > >>>>>>>>>>>>> 3. What does the value count for nested columns? > >>>> (Relevant: > >>>>>>>>>>>>> https://github.com/apache/parquet-format/issues/476) > >>>>>>>>>>>>> > >>>>>>>>>>>>> It seems to me that an exact count per row group is > >> not > >>>> very > >>>>>>>>> useful: > >>>>>>>>>>> since > >>>>>>>>>>>>> it cannot be combined across row groups or files any > >>>> sort of > >>>>>>>>>>> optimization > >>>>>>>>>>>>> that attempts to use it for a query like `select > >>>> count(distinct > >>>>>>>>> col) > >>>>>>>>>>> from > >>>>>>>>>>>>> t` breaks for many queries (also any query with a > >>>> filter, not > >>>>>>> to > >>>>>>>>>>> mention > >>>>>>>>>>>>> group by, etc.). We could do what Iceberg seems to do > >>>> and store > >>>>>>>>>>> sketches > >>>>>>>>>>>>> that can be combined across files/row groups, but I > >>>> think that > >>>>>>>>> should > >>>>>>>>>>> be a > >>>>>>>>>>>>> separate discussion as it would involve new fields. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Where this does seem like it could be useful is > >>>> optimizers > >>>>>>> (e.g. > >>>>>>>>> join > >>>>>>>>>>>>> ordering). For those applications I think an inexact > >>>> value is > >>>>>>>> just > >>>>>>>>> as > >>>>>>>>>>> good > >>>>>>>>>>>>> as an exact one. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thus I would like to propose that we clarify the > >> spec to > >>>> mark > >>>>>>>> this > >>>>>>>>>>> field > >>>>>>>>>>>>> as an estimate that can be off in either direction > >> by an > >>>>>>>> unbounded > >>>>>>>>>>> amount. > >>>>>>>>>>>>> If there are good use cases for error estimates we > >> could > >>>> add > >>>>>>> that > >>>>>>>>>>> later, > >>>>>>>>>>>>> and if we do we can get back exact values by setting > >> the > >>>> error > >>>>>>>> to 0 > >>>>>>>>>>>>> explicitly. The change would be a (possibly) 1 line > >>>> change in > >>>>>>> the > >>>>>>>>>>> comment > >>>>>>>>>>>>> of the thrift definition above. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Below is some investigation I did with an agent to > >> come > >>>> to this > >>>>>>>>>>> conclusion. > >>>>>>>>>>>>> > >>>>>>>>>>>>> ## History > >>>>>>>>>>>>> > >>>>>>>>>>>>> - 2013 (commit `2c4ada8`): Field 4 was > >> `list<ValueStats> > >>>>>>>>>>> distinct_values`. > >>>>>>>>>>>>> This was a list of all distinct values and the count > >> of > >>>> each > >>>>>>>> value. > >>>>>>>>>>>>> - 2013 (commit `d9039d0`): The field changed to `i64 > >>>>>>>>> distinct_count`. > >>>>>>>>>>> The > >>>>>>>>>>>>> commit message is "distinct values can be fetched > >> from > >>>> the > >>>>>>>>> dictionary > >>>>>>>>>>>>> page". This shows that the original design was an > >> exact > >>>> count > >>>>>>> of > >>>>>>>>> the > >>>>>>>>> > >> [message truncated...] > >
