Adrian, I think this issue goes back to Andrew's initial response, which I agree with:
However, I think redefining the existing distinct_count statistic > would have all the forward compatibility problems we have been debating Looking at existing usage and leveraging that to argue changing the meaning of a field undermines the guarantees we try to provide as a format. I can understand clarifying the definition, but this feels like we're trying to convince ourselves it's ok to redefine the field. There are a few scenarios we've discussed where it's reasonable that value could be ambiguous, but I think generally the expectation should be that it is accurate. If we think there's a better use case for carrying estimated values, maybe that's good justification for adding a new field (though with the limited usage you identified, maybe it's not worth the effort). -Dan On Thu, Oct 15, 2026 at 1:43 PM Adrian Garcia Badaracco <[email protected]> wrote: > To answer Micah's question, I looked at what current implementations do > with distinct_count, and how engines use NDVs from other sources. > I am familiar with DataFusion/arrow-rs, but for other systems (DuckDB, > Polars, Arrow C++, parquet-java, Spark, Trino, Iceberg, Hive and Impala) I > used an LLM to search their current implementation and did not verify the > findings by hand. > Please correct me if I got any of them wrong. > > Who writes distinct_count: > - arrow-rs: opt-in, off by default. It counts a set of value hashes. > - DuckDB: only for dictionary-encoded chunks (it writes the dictionary > size). > - Polars: only for boolean columns. > - parquet-java, Spark, Trino, Iceberg, Hive and Impala: never. > > Who reads it: > - DataFusion and Polars, and only as a planner estimate. > - No reader I found uses it for correctness. > > Current values are already not exact: > - Signed zero: arrow-rs and DuckDB both count -0.0 and +0.0 as two values. > - NaN: arrow-rs compares raw bytes, so NaNs with different bit patterns > count as different values. > - Hash collisions: arrow-rs counts xxHash64 hashes, not values, so > collisions can make the count too low. > - Dictionary arrays: arrow-rs counts dictionary keys, not values. Arrow > doesn't require dictionary values to be unique or used, so the count can be > too high. > So current writers can be wrong in both directions, and they don't share a > definition of equality. > > Engines with their own NDVs use estimates: > - Spark: HLL++, 5% default error [1] > - Trino: approx_distinct (HLL, about 2.3% error) for Hive tables [2], and > theta sketches for Iceberg tables > - Iceberg: Puffin theta sketches [3] > - Hive: HLL by default [4] > All of them use NDV only for cost-based planning: filter selectivity, join > cardinality, and through those, join order and broadcast choice. > > In conclsion: > - Readers don't depend on distinct_count being exact. > - Engines that compute their own NDVs choose estimates on purpose. > - So I think documenting the field as an estimate matches how it is used > today and covers most of the potential future use cases. > > [1] > https://github.com/apache/spark/blob/41f1694f7ba7cc3dd247d00dc42d13f5a8b1eaf7/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala#L399 > [2] > https://github.com/apache/trino/blob/cf8992f543d0671b2579fded1d17ae3ee862d543/core/trino-main/src/main/java/io/trino/operator/aggregation/DefaultApproximateCountDistinctAggregation.java#L41 > [3] > https://github.com/apache/iceberg/blob/acee558a84e8dd065bf49fa0af709a5346ede077/format/puffin-spec.md#L114-L123 > [4] > https://github.com/apache/hive/blob/da789295710cd04b7a76185c6551ec7ef5ab6f1f/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L2798-L2799 > > On 2026/10/15 20:08:22 Adrian Garcia Badaracco wrote: > > I sense the pushback is against saying that writers may use arbitrary > estimates. But unless we agree on how much error is acceptable, or on how > equality is defined, we are in effect allowing an unbounded estimate anyway. > > “Approximate” with no bound is the same as “arbitrary”. > > > > So I think we should pick one of these two options: > > > > 1. Define the allowed error (or the counting rules) precisely now. > > 2. Say explicitly that the value is an estimate with no guaranteed bound. > > > > I propose option 2 for now. We can come back and add tighter guarantees > when there are concrete use cases that need them, for example a reader that > needs to know the error stays within a specific bound, or that the writer > followed certain rules to compute the distinct count. > > Adding a guarantee later is backward compatible because existing readers > already treat the value as a hint. Removing one would not be. > > > > On 2026/10/07 19:52:38 Daniel Weeks wrote: > > > Andrew, I agree with you that the proposed wording is probably not the > best. > > > > > > I don't think the intent was to replace it with arbitrary estimates > (though > > > the current wording indicates that). > > > > > > I offered up some alternative language on the PR. > > > > > > -Dan > > > > > > > > > > > > On Wed, Oct 7, 2026 at 12:27 PM Andrew Lamb <[email protected]> > wrote: > > > > > > > IN my mind, the fact that "distinct" is not precisely defined for > all types > > > > means that changing "count of distinct values occurring" to > "Estimated > > > > count of distinct values occurring." is a clarification > > > > > > > > It seems like a significant change in meaning (explicitly allowing > > > > estimates) to me <shrug> > > > > > > > > Andrew > > > > > > > > On Wed, Oct 7, 2026 at 2:24 PM Daniel Weeks <[email protected]> > wrote: > > > > > > > > > I think the issue is less about whether this is changing something > for > > > > > older readers/writers or providing a more accurate description. > > > > > > > > > > Older readers would already "misinterpret" written values because > > > > there's a > > > > > difference between logical equivalence and physical equivalence. > This > > > > > difference is left up to interpretation by both the reader and > writer. > > > > > > > > > > An example would be -0 vs +0 for floating point types where some > systems > > > > > treat them equally where others may distinguish them. Similarly > NaN may > > > > or > > > > > may not be included as a value count. > > > > > > > > > > I think it's wrong to assume that current readers/writers are > capturing > > > > > this "correctly" when there is no "correct" or agreedupon > representation. > > > > > > > > > > -Dan > > > > > > > > > > On Wed, Oct 7, 2026 at 11:23 AM Russell Spitzer < > > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > My understanding is that the exactness of NDV is not > standardized at > > > > all > > > > > > for many types, which makes this value very difficult to define > or > > > > > > standardize as is. The main callouts during the meeting were > strings > > > > with > > > > > > collations (foo == Foo) floating point (NAN == NAN, -0.0 = 0.0 > etc...) > > > > So > > > > > > since we don't want to spend time addressing these issues, as > most use > > > > > > cases only require approximate statistics, the consensus was to > correct > > > > > the > > > > > > language now rather than spend time establishing something new. > Just a > > > > > > reminder for everyone, please make sure to add notes when > referring to > > > > > > discussions that occurred off list. I know the full video and > trascript > > > > > > will eventually be available but it's helpful to keep things > straight > > > > :) > > > > > > From the notes doc - > > > > > > > > > > > > - > > > > > > > > > > > > exactness of NDV stats: > > > > > > - > > > > > > > > > > > > Wanted to use in datafusion > > > > > > - > > > > > > > > > > > > Question of exactitude. > > > > > > - > > > > > > > > > > > > Collations makes it difficult because two distinct values > might > > > > be > > > > > > considered equal. > > > > > > - > > > > > > > > > > > > Semantics are under defined. > > > > > > - > > > > > > > > > > > > Do we want to formalize for floating point numbers: -0, > +0, > > > > > NaNs, > > > > > > etc? > > > > > > - > > > > > > > > > > > > Adrian: > > > > > > - > > > > > > > > > > > > Define right now as exact. > > > > > > - > > > > > > > > > > > > Most use cases do not require exact. > > > > > > - > > > > > > > > > > > > Statistical approximations of counts? > > > > > > - > > > > > > > > > > > > Kevin: > > > > > > - > > > > > > > > > > > > do we need to update the doc as well? > > > > > > - > > > > > > > > > > > > Yes! > > > > > > > > > > > > > > > > > > On Wed, Oct 7, 2026 at 1:08 PM Micah Kornfield < > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi Adrian, > > > > > > > Is there more context from the call, on why Andrew's assessment > > > > didn't > > > > > > make > > > > > > > sense? Are there reference implementations providing > approximate > > > > > answers > > > > > > > today? > > > > > > > > > > > > > > Thanks, > > > > > > > Micah > > > > > > > > > > > > > > On Wed, Oct 7, 2026 at 10:28 AM Adrian Garcia Badaracco < > > > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Per discussion on the sync call today I opened a format PR: > > > > > > > > https://github.com/apache/parquet-format/pull/628 > > > > > > > > > > > > > > > > On 2026/09/30 21:26:04 Andrew Lamb wrote: > > > > > > > > > I think optional approximate distinct (or other sketches > of the > > > > > data > > > > > > > like > > > > > > > > > histograms) would help various database planners make > better > > > > plans, > > > > > > so > > > > > > > > they > > > > > > > > > are valuable to explore. > > > > > > > > > > > > > > > > > > However, I think redefining the existing distinct_count > statistic > > > > > > > > > would have all the forward compatibility problems we have > been > > > > > > debating > > > > > > > > -- > > > > > > > > > namely we have no mechanism to prevent older readers > > > > > misinterpreting > > > > > > a > > > > > > > > > field whose semantic meaning was redefined. > > > > > > > > > > > > > > > > > > So maybe we could investigate adding new standardized > metadata > > > > > > instead > > > > > > > of > > > > > > > > > redefining the meaning of an old one > > > > > > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > On Tue, Sep 22, 2026 at 12:30 PM Adrian Garcia Badaracco > via dev > > > > < > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > The specification does not define what > > > > > `Statistics.distinct_count` > > > > > > > > > > contains. Writers do not know if they can write an > estimate. > > > > > > Readers > > > > > > > > do not > > > > > > > > > > know if they can use the value as an exact count. > > > > > > > > > > > > > > > > > > > > The field has this definition in `parquet.thrift`: > > > > > > > > > > > > > > > > > > > > ```thrift > > > > > > > > > > /** count of distinct values occurring */ > > > > > > > > > > 4: optional i64 distinct_count; > > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > > > The text does not answer these questions: > > > > > > > > > > > > > > > > > > > > 1. Must the value be exact? Or can it be an estimate, for > > > > example > > > > > > > from > > > > > > > > a > > > > > > > > > > HyperLogLog sketch? > > > > > > > > > > > > > > > > > > > > If we choose to make the value inexact, this opens up the > > > > > question: > > > > > > > > > > > > > > > > > > > > 1. If the value is not exact, what does it guarantee? Is > it an > > > > > > upper > > > > > > > > > > bound, a lower bound, or an estimate with no bound? > > > > > > > > > > > > > > > > > > > > If we choose to make the value exact, this opens up other > > > > > > questions: > > > > > > > > > > > > > > > > > > > > 1. How do nulls factor into the count? > > > > > > > > > > 2. How do writers compare values? For example, are > `-0.0` and > > > > > > `+0.0` > > > > > > > > one > > > > > > > > > > value or two? Are two NaN values with different bit > patterns > > > > one > > > > > > > value > > > > > > > > or > > > > > > > > > > two? Is the comparison on the physical bytes or on the > logical > > > > > > type? > > > > > > > > > > 3. What does the value count for nested columns? > (Relevant: > > > > > > > > > > https://github.com/apache/parquet-format/issues/476) > > > > > > > > > > > > > > > > > > > > It seems to me that an exact count per row group is not > very > > > > > > useful: > > > > > > > > since > > > > > > > > > > it cannot be combined across row groups or files any > sort of > > > > > > > > optimization > > > > > > > > > > that attempts to use it for a query like `select > count(distinct > > > > > > col) > > > > > > > > from > > > > > > > > > > t` breaks for many queries (also any query with a > filter, not > > > > to > > > > > > > > mention > > > > > > > > > > group by, etc.). We could do what Iceberg seems to do > and store > > > > > > > > sketches > > > > > > > > > > that can be combined across files/row groups, but I > think that > > > > > > should > > > > > > > > be a > > > > > > > > > > separate discussion as it would involve new fields. > > > > > > > > > > > > > > > > > > > > Where this does seem like it could be useful is > optimizers > > > > (e.g. > > > > > > join > > > > > > > > > > ordering). For those applications I think an inexact > value is > > > > > just > > > > > > as > > > > > > > > good > > > > > > > > > > as an exact one. > > > > > > > > > > > > > > > > > > > > Thus I would like to propose that we clarify the spec to > mark > > > > > this > > > > > > > > field > > > > > > > > > > as an estimate that can be off in either direction by an > > > > > unbounded > > > > > > > > amount. > > > > > > > > > > If there are good use cases for error estimates we could > add > > > > that > > > > > > > > later, > > > > > > > > > > and if we do we can get back exact values by setting the > error > > > > > to 0 > > > > > > > > > > explicitly. The change would be a (possibly) 1 line > change in > > > > the > > > > > > > > comment > > > > > > > > > > of the thrift definition above. > > > > > > > > > > > > > > > > > > > > Below is some investigation I did with an agent to come > to this > > > > > > > > conclusion. > > > > > > > > > > > > > > > > > > > > ## History > > > > > > > > > > > > > > > > > > > > - 2013 (commit `2c4ada8`): Field 4 was `list<ValueStats> > > > > > > > > distinct_values`. > > > > > > > > > > This was a list of all distinct values and the count of > each > > > > > value. > > > > > > > > > > - 2013 (commit `d9039d0`): The field changed to `i64 > > > > > > distinct_count`. > > > > > > > > The > > > > > > > > > > commit message is "distinct values can be fetched from > the > > > > > > dictionary > > > > > > > > > > page". This shows that the original design was an exact > count > > > > of > > > > > > the > > > > > > > > > > dictionary values. But the specification text did not > say this. > > > > > > > > > > - 2014 (commit `b2836e5`): The comment changed to the > current > > > > > text. > > > > > > > > There > > > > > > > > > > is no change in meaning since then. > > > > > > > > > > - 2023 (PARQUET-2352, #216): The specification changed > > > > > `min_value` > > > > > > > and > > > > > > > > > > `max_value` from exact values to "lower and upper bound > > > > values". > > > > > It > > > > > > > > also > > > > > > > > > > added `is_min_value_exact` and `is_max_value_exact`. This > > > > change > > > > > > did > > > > > > > > not > > > > > > > > > > include `distinct_count`. > > > > > > > > > > - 2024: A message on [email protected] asked if > > > > > > > `distinct_count` > > > > > > > > is > > > > > > > > > > exact, and if there are use cases for it. The thread > discussed > > > > > > > > `null_count` > > > > > > > > > > only. There was no answer about `distinct_count`. The > result > > > > was > > > > > > > #449, > > > > > > > > > > which also changed `null_count` only. Thread: > > > > > > > > > > > > > > https://lists.apache.org/thread/oqd9lt7p0jtf217hg0667xhk0zvwbgvt > > > > > > > > > > - 2014 to now: PARQUET-42 proposes HyperLogLog or > Count-Min > > > > > > sketches > > > > > > > as > > > > > > > > > > new statistics. This proposal is not active. > > > > > > > > > > > > > > > > > > > > ## Current implementations > > > > > > > > > > > > > > > > > > > > Few implementations write this field. No implementation > that we > > > > > > > > examined > > > > > > > > > > writes an estimate. There are known requests to write > > > > estimates, > > > > > > > > because an > > > > > > > > > > exact count for one row group has limited use for query > > > > planning > > > > > > (for > > > > > > > > > > example, https://github.com/apache/arrow-rs/issues/8608 > ). > > > > > > > > > > > > > > > > > > > > ### Writers > > > > > > > > > > > > > > > > > > > > | Implementation | Writes `distinct_count`? | Value | > > > > > > > > > > |---|---|---| > > > > > > > > > > | parquet-java | No | Not applicable | > > > > > > > > > > | Arrow C++ | Only if the user supplies a value | The > value > > > > from > > > > > > the > > > > > > > > user | > > > > > > > > > > | arrow-go | Only if the user supplies a value | The > value from > > > > > the > > > > > > > > user | > > > > > > > > > > | arrow-rs | Optional, off by default (since 2026-08) | > > > > Intended > > > > > as > > > > > > > > exact. > > > > > > > > > > It counts 64-bit hashes of the non-null values in the row > > > > group. > > > > > | > > > > > > > > > > | DuckDB | Yes, when the column chunk is > dictionary-encoded | > > > > The > > > > > > > > > > dictionary size | > > > > > > > > > > | Impala, cuDF, Polars | No | Not applicable | > > > > > > > > > > > > > > > > > > > > ### Readers > > > > > > > > > > > > > > > > > > > > | Implementation | Reads `distinct_count`? | Use | > > > > > > > > > > |---|---|---| > > > > > > > > > > | parquet-java and engines that use it (Spark, Trino, > Presto, > > > > > Hive, > > > > > > > > Drill) > > > > > > > > > > | No | Not applicable | > > > > > > > > > > | DuckDB | Yes | Shows the value in `parquet_metadata()` > only | > > > > > > > > > > | Polars | Yes | Cardinality estimation. It accepts an > > > > > approximate > > > > > > > > value. | > > > > > > > > > > | DataFusion | Yes | Cardinality estimation. If a file > has one > > > > > row > > > > > > > > group, > > > > > > > > > > it also uses the value as an exact answer for > `COUNT(DISTINCT > > > > > > col)`. > > > > > > > | > > > > > > > > > > > > > > > > > > > > ### Comparison: Apache Iceberg > > > > > > > > > > > > > > > > > > > > Iceberg keeps distinct-value statistics in Puffin files > as > > > > Apache > > > > > > > > > > DataSketches theta sketches. The Puffin specification > says that > > > > > the > > > > > > > > `ndv` > > > > > > > > > > property is an "estimate of number of distinct values, > derived > > > > > from > > > > > > > the > > > > > > > > > > sketch". Iceberg keeps these estimates separate from its > exact > > > > > file > > > > > > > > > > statistics. > > > > > > > > > > > > > > > > > > > > ## Possible solutions > > > > > > > > > > > > > > > > > > > > - Option A (exact): State that distinct_count MUST be > the exact > > > > > > > number > > > > > > > > of > > > > > > > > > > distinct values. > > > > > > > > > > - Option B (estimate): State that distinct_count is an > > > > estimate, > > > > > > > which > > > > > > > > can > > > > > > > > > > be higher or lower than the actual number by an amount > that the > > > > > > > > > > specification does not limit. > > > > > > > > > > - Option C (flag): Add an is_distinct_count_exact flag, > similar > > > > > to > > > > > > > > > > is_min_value_exact and is_max_value_exact. > > > > > > > > > > Extension to Options B and C (error bounds): Add optional > > > > fields > > > > > > for > > > > > > > a > > > > > > > > > > lower bound, an upper bound and the confidence level of > the > > > > > bounds. > > > > > > > > > > - Option D (separate field): Keep distinct_count exact > and add > > > > a > > > > > > new > > > > > > > > field > > > > > > > > > > for an estimate or for a sketch that readers can merge, > for > > > > > example > > > > > > > the > > > > > > > > > > Apache DataSketches format that Iceberg uses. > > > > > > > > > > - Option E (deprecate): Deprecate the field. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
