Makes sense. Thanks for helping bring this to a reasonable conclusion. I agree with your proposed paths forward.
I’ll open a new thread to propose introducing a new field to decouple this from a versioning conversation. Thanks all! > On Nov 10, 2026, at 5:36 PM, Daniel Weeks via dev <[email protected]> > wrote: > > I feel like we're reading into this too much to justify changing the > definition. I was involved in the project when this was introduced (over > 13 years ago), but I have no recollection of discussing the kinds of > concerns we're raising now. Issues like character uniqueness, equivalence, > estimates were very unlikely to have been considered at the time. However, > the lack of a clear statement of fact that the value is expected to be > exact, is no justification to assume that an estimate is a reasonable > replacement. > > There are a lot of issues with using distinct_count, for example, it's not > combinable across row groups or files, many implementations don't produce > it and likely fewer consume it. However, none of that is justification in > light of what it currently states and is expected to contain. > > As I see it, there are only two paths forward as I previously stated: 1) > introduce a new field, or 2) change the definition in conjunction with a > forward compatibility change (as discussed in the versioning topic). > > -Dan > > On Tue, Nov 10, 2026 at 4:06 PM Adrian Garcia Badaracco via dev < > [email protected]> wrote: > >> Hi Dan, >> >> Maybe I’m just missing context here. Were you part of introducing the >> field? Could you point me at info on where this field was ever understood >> to be exact? That might help me see the point of view that there was any >> expectation of this field being exact. >> >> Adrian >> >>> On Nov 10, 2026, at 12:30 PM, Daniel Weeks via dev < >> [email protected]> wrote: >>> >>> Hey Adrian, >>> >>> I'm ok with a clarification, but I don't think updating the definition to >>> be an estimate is inline with that. >>> >>> At this point, it feels like we're trying to justify changing the >>> definition (it was never clearly specified, it was never defined as >>> accurate, etc.) as opposed to looking at the intent behind the >>> introduction. It's very likely that we didn't understand the nuance >>> or complications at the time, but I don't think anyone understood it to >> be >>> an estimate. >>> >>> If we want to clarify it may not be accurate and could cause incorrect >>> results if relied on under certain cases, that would fall within a >>> compatible change. If we want to make an estimate, I believe it would >> have >>> to be a new field. >>> >>> It would be "easier" to just redefine it, but that's not compatible with >>> the guarantees we need to provide. That leaves us with creating a new >>> field if we want a value understood to be an estimate. >>> >>> The only reasonable alternative would be to wait until we agree on the >> next >>> major version at which point we could redefine the value (since a version >>> bump allows for backward incompatible changes like the proposed footer). >>> >>> Dan >>> >>> >>> >>> >>> On Mon, Nov 9, 2026 at 7:20 AM Adrian Garcia Badaracco < >> [email protected]> >>> wrote: >>> >>>> Hi Dan, >>>> >>>> I understand the concern about redefining a field. I see this as >> defining >>>> the field for the first time. The field was never defined as accurate, >>>> accuracy was simply never specified. No reader expects it to be >> accurate, >>>> and no writer writes completely accurate values. >>>> >>>> There is precedent for a similar change for zone map min/max stats: in >>>> 2023, PARQUET-2352 (#216 < >>>> https://github.com/apache/parquet-format/pull/216>) changed `min_value` >>>> and `max_value` from "Min and max values" to "Lower and upper bound >> values". >>>> >>>> If you still prefer not to change the text of `distinct_count`, I am >> fine >>>> with the alternative: deprecate `distinct_count` and add a new field. I >>>> would propose that the new field be `estimated_distinct_count`, defined >> as >>>> “An estimate of the number of distinct non-null values in the page or >>>> column chunk. The estimate can be higher or lower than the actual count >> by >>>> any amount.". This field would be useful for join ordering estimation, >>>> which is the use case multiple systems are using the existing field >> for. It >>>> would be easy and performant to compute and use for both readers and >>>> writers, and the harm when it is grossly wrong is limited. >>>> >>>> The main alternative I can think of is a sketch. A sketch can be merged >>>> across row groups and files, which helps for single-column joins. But >> it is >>>> larger, more complex to define, and needs agreement on a sketch format. >> It >>>> also does not help much for joins on multiple columns and other more >>>> complex cases. So I think starting with a simple per-column per-chunk >>>> estimated distinct count integer and adding a sketch later if there are >>>> good use cases for it would be the better path forward. >>>> >>>> Do you see us agreeing on one of these two paths (update the definition >> to >>>> be an estimate or deprecate and add a new field)? >>>> >>>> Thanks, >>>> >>>> Adrian >>>> >>>> On 2026/10/19 14:33:14 Daniel Weeks wrote: >>>>> Adrian, I think this issue goes back to Andrew's initial response, >> which >>>> I >>>>> agree with: >>>>> >>>>> However, I think redefining the existing distinct_count statistic >>>>>> would have all the forward compatibility problems we have been >> debating >>>>> >>>>> >>>>> Looking at existing usage and leveraging that to argue changing the >>>> meaning >>>>> of a field undermines the guarantees we try to provide as a format. >>>>> >>>>> I can understand clarifying the definition, but this feels like we're >>>>> trying to convince ourselves it's ok to redefine the field. >>>>> >>>>> There are a few scenarios we've discussed where it's reasonable that >>>> value >>>>> could be ambiguous, but I think generally the expectation should be >> that >>>> it >>>>> is accurate. >>>>> >>>>> If we think there's a better use case for carrying estimated values, >>>> maybe >>>>> that's good justification for adding a new field (though with the >> limited >>>>> usage you identified, maybe it's not worth the effort). >>>>> >>>>> -Dan >>>>> >>>>> On Thu, Oct 15, 2026 at 1:43 PM Adrian Garcia Badaracco < >>>> [email protected]> >>>>> wrote: >>>>> >>>>>> To answer Micah's question, I looked at what current implementations >> do >>>>>> with distinct_count, and how engines use NDVs from other sources. >>>>>> I am familiar with DataFusion/arrow-rs, but for other systems (DuckDB, >>>>>> Polars, Arrow C++, parquet-java, Spark, Trino, Iceberg, Hive and >>>> Impala) I >>>>>> used an LLM to search their current implementation and did not verify >>>> the >>>>>> findings by hand. >>>>>> Please correct me if I got any of them wrong. >>>>>> >>>>>> Who writes distinct_count: >>>>>> - arrow-rs: opt-in, off by default. It counts a set of value hashes. >>>>>> - DuckDB: only for dictionary-encoded chunks (it writes the dictionary >>>>>> size). >>>>>> - Polars: only for boolean columns. >>>>>> - parquet-java, Spark, Trino, Iceberg, Hive and Impala: never. >>>>>> >>>>>> Who reads it: >>>>>> - DataFusion and Polars, and only as a planner estimate. >>>>>> - No reader I found uses it for correctness. >>>>>> >>>>>> Current values are already not exact: >>>>>> - Signed zero: arrow-rs and DuckDB both count -0.0 and +0.0 as two >>>> values. >>>>>> - NaN: arrow-rs compares raw bytes, so NaNs with different bit >> patterns >>>>>> count as different values. >>>>>> - Hash collisions: arrow-rs counts xxHash64 hashes, not values, so >>>>>> collisions can make the count too low. >>>>>> - Dictionary arrays: arrow-rs counts dictionary keys, not values. >> Arrow >>>>>> doesn't require dictionary values to be unique or used, so the count >>>> can be >>>>>> too high. >>>>>> So current writers can be wrong in both directions, and they don't >>>> share a >>>>>> definition of equality. >>>>>> >>>>>> Engines with their own NDVs use estimates: >>>>>> - Spark: HLL++, 5% default error [1] >>>>>> - Trino: approx_distinct (HLL, about 2.3% error) for Hive tables [2], >>>> and >>>>>> theta sketches for Iceberg tables >>>>>> - Iceberg: Puffin theta sketches [3] >>>>>> - Hive: HLL by default [4] >>>>>> All of them use NDV only for cost-based planning: filter selectivity, >>>> join >>>>>> cardinality, and through those, join order and broadcast choice. >>>>>> >>>>>> In conclsion: >>>>>> - Readers don't depend on distinct_count being exact. >>>>>> - Engines that compute their own NDVs choose estimates on purpose. >>>>>> - So I think documenting the field as an estimate matches how it is >>>> used >>>>>> today and covers most of the potential future use cases. >>>>>> >>>>>> [1] >>>>>> >>>> >> https://github.com/apache/spark/blob/41f1694f7ba7cc3dd247d00dc42d13f5a8b1eaf7/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala#L399 >>>>>> [2] >>>>>> >>>> >> https://github.com/apache/trino/blob/cf8992f543d0671b2579fded1d17ae3ee862d543/core/trino-main/src/main/java/io/trino/operator/aggregation/DefaultApproximateCountDistinctAggregation.java#L41 >>>>>> [3] >>>>>> >>>> >> https://github.com/apache/iceberg/blob/acee558a84e8dd065bf49fa0af709a5346ede077/format/puffin-spec.md#L114-L123 >>>>>> [4] >>>>>> >>>> >> https://github.com/apache/hive/blob/da789295710cd04b7a76185c6551ec7ef5ab6f1f/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L2798-L2799 >>>>>> >>>>>> On 2026/10/15 20:08:22 Adrian Garcia Badaracco wrote: >>>>>>> I sense the pushback is against saying that writers may use arbitrary >>>>>> estimates. But unless we agree on how much error is acceptable, or on >>>> how >>>>>> equality is defined, we are in effect allowing an unbounded estimate >>>> anyway. >>>>>>> “Approximate” with no bound is the same as “arbitrary”. >>>>>>> >>>>>>> So I think we should pick one of these two options: >>>>>>> >>>>>>> 1. Define the allowed error (or the counting rules) precisely now. >>>>>>> 2. Say explicitly that the value is an estimate with no guaranteed >>>> bound. >>>>>>> >>>>>>> I propose option 2 for now. We can come back and add tighter >>>> guarantees >>>>>> when there are concrete use cases that need them, for example a reader >>>> that >>>>>> needs to know the error stays within a specific bound, or that the >>>> writer >>>>>> followed certain rules to compute the distinct count. >>>>>>> Adding a guarantee later is backward compatible because existing >>>> readers >>>>>> already treat the value as a hint. Removing one would not be. >>>>>>> >>>>>>> On 2026/10/07 19:52:38 Daniel Weeks wrote: >>>>>>>> Andrew, I agree with you that the proposed wording is probably not >>>> the >>>>>> best. >>>>>>>> >>>>>>>> I don't think the intent was to replace it with arbitrary estimates >>>>>> (though >>>>>>>> the current wording indicates that). >>>>>>>> >>>>>>>> I offered up some alternative language on the PR. >>>>>>>> >>>>>>>> -Dan >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Oct 7, 2026 at 12:27 PM Andrew Lamb <[email protected]> >>>>>> wrote: >>>>>>>> >>>>>>>>> IN my mind, the fact that "distinct" is not precisely defined for >>>>>> all types >>>>>>>>> means that changing "count of distinct values occurring" to >>>>>> "Estimated >>>>>>>>> count of distinct values occurring." is a clarification >>>>>>>>> >>>>>>>>> It seems like a significant change in meaning (explicitly >>>> allowing >>>>>>>>> estimates) to me <shrug> >>>>>>>>> >>>>>>>>> Andrew >>>>>>>>> >>>>>>>>> On Wed, Oct 7, 2026 at 2:24 PM Daniel Weeks <[email protected]> >>>>>> wrote: >>>>>>>>> >>>>>>>>>> I think the issue is less about whether this is changing >>>> something >>>>>> for >>>>>>>>>> older readers/writers or providing a more accurate description. >>>>>>>>>> >>>>>>>>>> Older readers would already "misinterpret" written values >>>> because >>>>>>>>> there's a >>>>>>>>>> difference between logical equivalence and physical >>>> equivalence. >>>>>> This >>>>>>>>>> difference is left up to interpretation by both the reader and >>>>>> writer. >>>>>>>>>> >>>>>>>>>> An example would be -0 vs +0 for floating point types where >>>> some >>>>>> systems >>>>>>>>>> treat them equally where others may distinguish them. >>>> Similarly >>>>>> NaN may >>>>>>>>> or >>>>>>>>>> may not be included as a value count. >>>>>>>>>> >>>>>>>>>> I think it's wrong to assume that current readers/writers are >>>>>> capturing >>>>>>>>>> this "correctly" when there is no "correct" or agreedupon >>>>>> representation. >>>>>>>>>> >>>>>>>>>> -Dan >>>>>>>>>> >>>>>>>>>> On Wed, Oct 7, 2026 at 11:23 AM Russell Spitzer < >>>>>>>>> [email protected] >>>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> My understanding is that the exactness of NDV is not >>>>>> standardized at >>>>>>>>> all >>>>>>>>>>> for many types, which makes this value very difficult to >>>> define >>>>>> or >>>>>>>>>>> standardize as is. The main callouts during the meeting were >>>>>> strings >>>>>>>>> with >>>>>>>>>>> collations (foo == Foo) floating point (NAN == NAN, -0.0 = >>>> 0.0 >>>>>> etc...) >>>>>>>>> So >>>>>>>>>>> since we don't want to spend time addressing these issues, as >>>>>> most use >>>>>>>>>>> cases only require approximate statistics, the consensus was >>>> to >>>>>> correct >>>>>>>>>> the >>>>>>>>>>> language now rather than spend time establishing something >>>> new. >>>>>> Just a >>>>>>>>>>> reminder for everyone, please make sure to add notes when >>>>>> referring to >>>>>>>>>>> discussions that occurred off list. I know the full video and >>>>>> trascript >>>>>>>>>>> will eventually be available but it's helpful to keep things >>>>>> straight >>>>>>>>> :) >>>>>>>>>>> From the notes doc - >>>>>>>>>>> >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> exactness of NDV stats: >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Wanted to use in datafusion >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Question of exactitude. >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Collations makes it difficult because two distinct >>>> values >>>>>> might >>>>>>>>> be >>>>>>>>>>> considered equal. >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Semantics are under defined. >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Do we want to formalize for floating point numbers: >>>> -0, >>>>>> +0, >>>>>>>>>> NaNs, >>>>>>>>>>> etc? >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Adrian: >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Define right now as exact. >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Most use cases do not require exact. >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Statistical approximations of counts? >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Kevin: >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> do we need to update the doc as well? >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>>>> Yes! >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Oct 7, 2026 at 1:08 PM Micah Kornfield < >>>>>> [email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Adrian, >>>>>>>>>>>> Is there more context from the call, on why Andrew's >>>> assessment >>>>>>>>> didn't >>>>>>>>>>> make >>>>>>>>>>>> sense? Are there reference implementations providing >>>>>> approximate >>>>>>>>>> answers >>>>>>>>>>>> today? >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Micah >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Oct 7, 2026 at 10:28 AM Adrian Garcia Badaracco < >>>>>>>>>>>> [email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Per discussion on the sync call today I opened a format >>>> PR: >>>>>>>>>>>>> https://github.com/apache/parquet-format/pull/628 >>>>>>>>>>>>> >>>>>>>>>>>>> On 2026/09/30 21:26:04 Andrew Lamb wrote: >>>>>>>>>>>>>> I think optional approximate distinct (or other >>>> sketches >>>>>> of the >>>>>>>>>> data >>>>>>>>>>>> like >>>>>>>>>>>>>> histograms) would help various database planners make >>>>>> better >>>>>>>>> plans, >>>>>>>>>>> so >>>>>>>>>>>>> they >>>>>>>>>>>>>> are valuable to explore. >>>>>>>>>>>>>> >>>>>>>>>>>>>> However, I think redefining the existing distinct_count >>>>>> statistic >>>>>>>>>>>>>> would have all the forward compatibility problems we >>>> have >>>>>> been >>>>>>>>>>> debating >>>>>>>>>>>>> -- >>>>>>>>>>>>>> namely we have no mechanism to prevent older readers >>>>>>>>>> misinterpreting >>>>>>>>>>> a >>>>>>>>>>>>>> field whose semantic meaning was redefined. >>>>>>>>>>>>>> >>>>>>>>>>>>>> So maybe we could investigate adding new standardized >>>>>> metadata >>>>>>>>>>> instead >>>>>>>>>>>> of >>>>>>>>>>>>>> redefining the meaning of an old one >>>>>>>>>>>>>> >>>>>>>>>>>>>> Andrew >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Sep 22, 2026 at 12:30 PM Adrian Garcia >>>> Badaracco >>>>>> via dev >>>>>>>>> < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> The specification does not define what >>>>>>>>>> `Statistics.distinct_count` >>>>>>>>>>>>>>> contains. Writers do not know if they can write an >>>>>> estimate. >>>>>>>>>>> Readers >>>>>>>>>>>>> do not >>>>>>>>>>>>>>> know if they can use the value as an exact count. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The field has this definition in `parquet.thrift`: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ```thrift >>>>>>>>>>>>>>> /** count of distinct values occurring */ >>>>>>>>>>>>>>> 4: optional i64 distinct_count; >>>>>>>>>>>>>>> ``` >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The text does not answer these questions: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. Must the value be exact? Or can it be an >>>> estimate, for >>>>>>>>> example >>>>>>>>>>>> from >>>>>>>>>>>>> a >>>>>>>>>>>>>>> HyperLogLog sketch? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If we choose to make the value inexact, this opens >>>> up the >>>>>>>>>> question: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. If the value is not exact, what does it >>>> guarantee? Is >>>>>> it an >>>>>>>>>>> upper >>>>>>>>>>>>>>> bound, a lower bound, or an estimate with no bound? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If we choose to make the value exact, this opens up >>>> other >>>>>>>>>>> questions: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. How do nulls factor into the count? >>>>>>>>>>>>>>> 2. How do writers compare values? For example, are >>>>>> `-0.0` and >>>>>>>>>>> `+0.0` >>>>>>>>>>>>> one >>>>>>>>>>>>>>> value or two? Are two NaN values with different bit >>>>>> patterns >>>>>>>>> one >>>>>>>>>>>> value >>>>>>>>>>>>> or >>>>>>>>>>>>>>> two? Is the comparison on the physical bytes or on >>>> the >>>>>> logical >>>>>>>>>>> type? >>>>>>>>>>>>>>> 3. What does the value count for nested columns? >>>>>> (Relevant: >>>>>>>>>>>>>>> https://github.com/apache/parquet-format/issues/476) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It seems to me that an exact count per row group is >>>> not >>>>>> very >>>>>>>>>>> useful: >>>>>>>>>>>>> since >>>>>>>>>>>>>>> it cannot be combined across row groups or files any >>>>>> sort of >>>>>>>>>>>>> optimization >>>>>>>>>>>>>>> that attempts to use it for a query like `select >>>>>> count(distinct >>>>>>>>>>> col) >>>>>>>>>>>>> from >>>>>>>>>>>>>>> t` breaks for many queries (also any query with a >>>>>> filter, not >>>>>>>>> to >>>>>>>>>>>>> mention >>>>>>>>>>>>>>> group by, etc.). We could do what Iceberg seems to do >>>>>> and store >>>>>>>>>>>>> sketches >>>>>>>>>>>>>>> that can be combined across files/row groups, but I >>>>>> think that >>>>>>>>>>> should >>>>>>>>>>>>> be a >>>>>>>>>>>>>>> separate discussion as it would involve new fields. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Where this does seem like it could be useful is >>>>>> optimizers >>>>>>>>> (e.g. >>>>>>>>>>> join >>>>>>>>>>>>>>> ordering). For those applications I think an inexact >>>>>> value is >>>>>>>>>> just >>>>>>>>>>> as >>>>>>>>>>>>> good >>>>>>>>>>>>>>> as an exact one. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thus I would like to propose that we clarify the >>>> spec to >>>>>> mark >>>>>>>>>> this >>>>>>>>>>>>> field >>>>>>>>>>>>>>> as an estimate that can be off in either direction >>>> by an >>>>>>>>>> unbounded >>>>>>>>>>>>> amount. >>>>>>>>>>>>>>> If there are good use cases for error estimates we >>>> could >>>>>> add >>>>>>>>> that >>>>>>>>>>>>> later, >>>>>>>>>>>>>>> and if we do we can get back exact values by setting >>>> the >>>>>> error >>>>>>>>>> to 0 >>>>>>>>>>>>>>> explicitly. The change would be a (possibly) 1 line >>>>>> change in >>>>>>>>> the >>>>>>>>>>>>> comment >>>>>>>>>>>>>>> of the thrift definition above. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Below is some investigation I did with an agent to >>>> come >>>>>> to this >>>>>>>>>>>>> conclusion. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ## History >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - 2013 (commit `2c4ada8`): Field 4 was >>>> `list<ValueStats> >>>>>>>>>>>>> distinct_values`. >>>>>>>>>>>>>>> This was a list of all distinct values and the count >>>> of >>>>>> each >>>>>>>>>> value. >>>>>>>>>>>>>>> - 2013 (commit `d9039d0`): The field changed to `i64 >>>>>>>>>>> distinct_count`. >>>>>>>>>>>>> The >>>>>>>>>>>>>>> commit message is "distinct values can be fetched >>>> from >>>>>> the >>>>>>>>>>> dictionary >>>>>>>>>>>>>>> page". This shows that the original design was an >>>> exact >>>>>> count >>>>>>>>> of >>>>>>>>>>> the >>>>>>>>>>> >>>> [message truncated...] >> >>
