I sense the pushback is against saying that writers may use arbitrary estimates. But unless we agree on how much error is acceptable, or on how equality is defined, we are in effect allowing an unbounded estimate anyway. “Approximate” with no bound is the same as “arbitrary”.
So I think we should pick one of these two options: 1. Define the allowed error (or the counting rules) precisely now. 2. Say explicitly that the value is an estimate with no guaranteed bound. I propose option 2 for now. We can come back and add tighter guarantees when there are concrete use cases that need them, for example a reader that needs to know the error stays within a specific bound, or that the writer followed certain rules to compute the distinct count. Adding a guarantee later is backward compatible because existing readers already treat the value as a hint. Removing one would not be. On 2026/10/07 19:52:38 Daniel Weeks wrote: > Andrew, I agree with you that the proposed wording is probably not the best. > > I don't think the intent was to replace it with arbitrary estimates (though > the current wording indicates that). > > I offered up some alternative language on the PR. > > -Dan > > > > On Wed, Oct 7, 2026 at 12:27 PM Andrew Lamb <[email protected]> wrote: > > > IN my mind, the fact that "distinct" is not precisely defined for all types > > means that changing "count of distinct values occurring" to "Estimated > > count of distinct values occurring." is a clarification > > > > It seems like a significant change in meaning (explicitly allowing > > estimates) to me <shrug> > > > > Andrew > > > > On Wed, Oct 7, 2026 at 2:24 PM Daniel Weeks <[email protected]> wrote: > > > > > I think the issue is less about whether this is changing something for > > > older readers/writers or providing a more accurate description. > > > > > > Older readers would already "misinterpret" written values because > > there's a > > > difference between logical equivalence and physical equivalence. This > > > difference is left up to interpretation by both the reader and writer. > > > > > > An example would be -0 vs +0 for floating point types where some systems > > > treat them equally where others may distinguish them. Similarly NaN may > > or > > > may not be included as a value count. > > > > > > I think it's wrong to assume that current readers/writers are capturing > > > this "correctly" when there is no "correct" or agreedupon representation. > > > > > > -Dan > > > > > > On Wed, Oct 7, 2026 at 11:23 AM Russell Spitzer < > > [email protected] > > > > > > > wrote: > > > > > > > My understanding is that the exactness of NDV is not standardized at > > all > > > > for many types, which makes this value very difficult to define or > > > > standardize as is. The main callouts during the meeting were strings > > with > > > > collations (foo == Foo) floating point (NAN == NAN, -0.0 = 0.0 etc...) > > So > > > > since we don't want to spend time addressing these issues, as most use > > > > cases only require approximate statistics, the consensus was to correct > > > the > > > > language now rather than spend time establishing something new. Just a > > > > reminder for everyone, please make sure to add notes when referring to > > > > discussions that occurred off list. I know the full video and trascript > > > > will eventually be available but it's helpful to keep things straight > > :) > > > > From the notes doc - > > > > > > > > - > > > > > > > > exactness of NDV stats: > > > > - > > > > > > > > Wanted to use in datafusion > > > > - > > > > > > > > Question of exactitude. > > > > - > > > > > > > > Collations makes it difficult because two distinct values might > > be > > > > considered equal. > > > > - > > > > > > > > Semantics are under defined. > > > > - > > > > > > > > Do we want to formalize for floating point numbers: -0, +0, > > > NaNs, > > > > etc? > > > > - > > > > > > > > Adrian: > > > > - > > > > > > > > Define right now as exact. > > > > - > > > > > > > > Most use cases do not require exact. > > > > - > > > > > > > > Statistical approximations of counts? > > > > - > > > > > > > > Kevin: > > > > - > > > > > > > > do we need to update the doc as well? > > > > - > > > > > > > > Yes! > > > > > > > > > > > > On Wed, Oct 7, 2026 at 1:08 PM Micah Kornfield <[email protected]> > > > > wrote: > > > > > > > > > Hi Adrian, > > > > > Is there more context from the call, on why Andrew's assessment > > didn't > > > > make > > > > > sense? Are there reference implementations providing approximate > > > answers > > > > > today? > > > > > > > > > > Thanks, > > > > > Micah > > > > > > > > > > On Wed, Oct 7, 2026 at 10:28 AM Adrian Garcia Badaracco < > > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > Per discussion on the sync call today I opened a format PR: > > > > > > https://github.com/apache/parquet-format/pull/628 > > > > > > > > > > > > On 2026/09/30 21:26:04 Andrew Lamb wrote: > > > > > > > I think optional approximate distinct (or other sketches of the > > > data > > > > > like > > > > > > > histograms) would help various database planners make better > > plans, > > > > so > > > > > > they > > > > > > > are valuable to explore. > > > > > > > > > > > > > > However, I think redefining the existing distinct_count statistic > > > > > > > would have all the forward compatibility problems we have been > > > > debating > > > > > > -- > > > > > > > namely we have no mechanism to prevent older readers > > > misinterpreting > > > > a > > > > > > > field whose semantic meaning was redefined. > > > > > > > > > > > > > > So maybe we could investigate adding new standardized metadata > > > > instead > > > > > of > > > > > > > redefining the meaning of an old one > > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > On Tue, Sep 22, 2026 at 12:30 PM Adrian Garcia Badaracco via dev > > < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > The specification does not define what > > > `Statistics.distinct_count` > > > > > > > > contains. Writers do not know if they can write an estimate. > > > > Readers > > > > > > do not > > > > > > > > know if they can use the value as an exact count. > > > > > > > > > > > > > > > > The field has this definition in `parquet.thrift`: > > > > > > > > > > > > > > > > ```thrift > > > > > > > > /** count of distinct values occurring */ > > > > > > > > 4: optional i64 distinct_count; > > > > > > > > ``` > > > > > > > > > > > > > > > > The text does not answer these questions: > > > > > > > > > > > > > > > > 1. Must the value be exact? Or can it be an estimate, for > > example > > > > > from > > > > > > a > > > > > > > > HyperLogLog sketch? > > > > > > > > > > > > > > > > If we choose to make the value inexact, this opens up the > > > question: > > > > > > > > > > > > > > > > 1. If the value is not exact, what does it guarantee? Is it an > > > > upper > > > > > > > > bound, a lower bound, or an estimate with no bound? > > > > > > > > > > > > > > > > If we choose to make the value exact, this opens up other > > > > questions: > > > > > > > > > > > > > > > > 1. How do nulls factor into the count? > > > > > > > > 2. How do writers compare values? For example, are `-0.0` and > > > > `+0.0` > > > > > > one > > > > > > > > value or two? Are two NaN values with different bit patterns > > one > > > > > value > > > > > > or > > > > > > > > two? Is the comparison on the physical bytes or on the logical > > > > type? > > > > > > > > 3. What does the value count for nested columns? (Relevant: > > > > > > > > https://github.com/apache/parquet-format/issues/476) > > > > > > > > > > > > > > > > It seems to me that an exact count per row group is not very > > > > useful: > > > > > > since > > > > > > > > it cannot be combined across row groups or files any sort of > > > > > > optimization > > > > > > > > that attempts to use it for a query like `select count(distinct > > > > col) > > > > > > from > > > > > > > > t` breaks for many queries (also any query with a filter, not > > to > > > > > > mention > > > > > > > > group by, etc.). We could do what Iceberg seems to do and store > > > > > > sketches > > > > > > > > that can be combined across files/row groups, but I think that > > > > should > > > > > > be a > > > > > > > > separate discussion as it would involve new fields. > > > > > > > > > > > > > > > > Where this does seem like it could be useful is optimizers > > (e.g. > > > > join > > > > > > > > ordering). For those applications I think an inexact value is > > > just > > > > as > > > > > > good > > > > > > > > as an exact one. > > > > > > > > > > > > > > > > Thus I would like to propose that we clarify the spec to mark > > > this > > > > > > field > > > > > > > > as an estimate that can be off in either direction by an > > > unbounded > > > > > > amount. > > > > > > > > If there are good use cases for error estimates we could add > > that > > > > > > later, > > > > > > > > and if we do we can get back exact values by setting the error > > > to 0 > > > > > > > > explicitly. The change would be a (possibly) 1 line change in > > the > > > > > > comment > > > > > > > > of the thrift definition above. > > > > > > > > > > > > > > > > Below is some investigation I did with an agent to come to this > > > > > > conclusion. > > > > > > > > > > > > > > > > ## History > > > > > > > > > > > > > > > > - 2013 (commit `2c4ada8`): Field 4 was `list<ValueStats> > > > > > > distinct_values`. > > > > > > > > This was a list of all distinct values and the count of each > > > value. > > > > > > > > - 2013 (commit `d9039d0`): The field changed to `i64 > > > > distinct_count`. > > > > > > The > > > > > > > > commit message is "distinct values can be fetched from the > > > > dictionary > > > > > > > > page". This shows that the original design was an exact count > > of > > > > the > > > > > > > > dictionary values. But the specification text did not say this. > > > > > > > > - 2014 (commit `b2836e5`): The comment changed to the current > > > text. > > > > > > There > > > > > > > > is no change in meaning since then. > > > > > > > > - 2023 (PARQUET-2352, #216): The specification changed > > > `min_value` > > > > > and > > > > > > > > `max_value` from exact values to "lower and upper bound > > values". > > > It > > > > > > also > > > > > > > > added `is_min_value_exact` and `is_max_value_exact`. This > > change > > > > did > > > > > > not > > > > > > > > include `distinct_count`. > > > > > > > > - 2024: A message on [email protected] asked if > > > > > `distinct_count` > > > > > > is > > > > > > > > exact, and if there are use cases for it. The thread discussed > > > > > > `null_count` > > > > > > > > only. There was no answer about `distinct_count`. The result > > was > > > > > #449, > > > > > > > > which also changed `null_count` only. Thread: > > > > > > > > > > https://lists.apache.org/thread/oqd9lt7p0jtf217hg0667xhk0zvwbgvt > > > > > > > > - 2014 to now: PARQUET-42 proposes HyperLogLog or Count-Min > > > > sketches > > > > > as > > > > > > > > new statistics. This proposal is not active. > > > > > > > > > > > > > > > > ## Current implementations > > > > > > > > > > > > > > > > Few implementations write this field. No implementation that we > > > > > > examined > > > > > > > > writes an estimate. There are known requests to write > > estimates, > > > > > > because an > > > > > > > > exact count for one row group has limited use for query > > planning > > > > (for > > > > > > > > example, https://github.com/apache/arrow-rs/issues/8608). > > > > > > > > > > > > > > > > ### Writers > > > > > > > > > > > > > > > > | Implementation | Writes `distinct_count`? | Value | > > > > > > > > |---|---|---| > > > > > > > > | parquet-java | No | Not applicable | > > > > > > > > | Arrow C++ | Only if the user supplies a value | The value > > from > > > > the > > > > > > user | > > > > > > > > | arrow-go | Only if the user supplies a value | The value from > > > the > > > > > > user | > > > > > > > > | arrow-rs | Optional, off by default (since 2026-08) | > > Intended > > > as > > > > > > exact. > > > > > > > > It counts 64-bit hashes of the non-null values in the row > > group. > > > | > > > > > > > > | DuckDB | Yes, when the column chunk is dictionary-encoded | > > The > > > > > > > > dictionary size | > > > > > > > > | Impala, cuDF, Polars | No | Not applicable | > > > > > > > > > > > > > > > > ### Readers > > > > > > > > > > > > > > > > | Implementation | Reads `distinct_count`? | Use | > > > > > > > > |---|---|---| > > > > > > > > | parquet-java and engines that use it (Spark, Trino, Presto, > > > Hive, > > > > > > Drill) > > > > > > > > | No | Not applicable | > > > > > > > > | DuckDB | Yes | Shows the value in `parquet_metadata()` only | > > > > > > > > | Polars | Yes | Cardinality estimation. It accepts an > > > approximate > > > > > > value. | > > > > > > > > | DataFusion | Yes | Cardinality estimation. If a file has one > > > row > > > > > > group, > > > > > > > > it also uses the value as an exact answer for `COUNT(DISTINCT > > > > col)`. > > > > > | > > > > > > > > > > > > > > > > ### Comparison: Apache Iceberg > > > > > > > > > > > > > > > > Iceberg keeps distinct-value statistics in Puffin files as > > Apache > > > > > > > > DataSketches theta sketches. The Puffin specification says that > > > the > > > > > > `ndv` > > > > > > > > property is an "estimate of number of distinct values, derived > > > from > > > > > the > > > > > > > > sketch". Iceberg keeps these estimates separate from its exact > > > file > > > > > > > > statistics. > > > > > > > > > > > > > > > > ## Possible solutions > > > > > > > > > > > > > > > > - Option A (exact): State that distinct_count MUST be the exact > > > > > number > > > > > > of > > > > > > > > distinct values. > > > > > > > > - Option B (estimate): State that distinct_count is an > > estimate, > > > > > which > > > > > > can > > > > > > > > be higher or lower than the actual number by an amount that the > > > > > > > > specification does not limit. > > > > > > > > - Option C (flag): Add an is_distinct_count_exact flag, similar > > > to > > > > > > > > is_min_value_exact and is_max_value_exact. > > > > > > > > Extension to Options B and C (error bounds): Add optional > > fields > > > > for > > > > > a > > > > > > > > lower bound, an upper bound and the confidence level of the > > > bounds. > > > > > > > > - Option D (separate field): Keep distinct_count exact and add > > a > > > > new > > > > > > field > > > > > > > > for an estimate or for a sketch that readers can merge, for > > > example > > > > > the > > > > > > > > Apache DataSketches format that Iceberg uses. > > > > > > > > - Option E (deprecate): Deprecate the field. > > > > > > > > > > > > > > > > > > > > > > > > > > > >
