To answer Micah's question, I looked at what current implementations do with distinct_count, and how engines use NDVs from other sources. I am familiar with DataFusion/arrow-rs, but for other systems (DuckDB, Polars, Arrow C++, parquet-java, Spark, Trino, Iceberg, Hive and Impala) I used an LLM to search their current implementation and did not verify the findings by hand. Please correct me if I got any of them wrong.
Who writes distinct_count: - arrow-rs: opt-in, off by default. It counts a set of value hashes. - DuckDB: only for dictionary-encoded chunks (it writes the dictionary size). - Polars: only for boolean columns. - parquet-java, Spark, Trino, Iceberg, Hive and Impala: never. Who reads it: - DataFusion and Polars, and only as a planner estimate. - No reader I found uses it for correctness. Current values are already not exact: - Signed zero: arrow-rs and DuckDB both count -0.0 and +0.0 as two values. - NaN: arrow-rs compares raw bytes, so NaNs with different bit patterns count as different values. - Hash collisions: arrow-rs counts xxHash64 hashes, not values, so collisions can make the count too low. - Dictionary arrays: arrow-rs counts dictionary keys, not values. Arrow doesn't require dictionary values to be unique or used, so the count can be too high. So current writers can be wrong in both directions, and they don't share a definition of equality. Engines with their own NDVs use estimates: - Spark: HLL++, 5% default error [1] - Trino: approx_distinct (HLL, about 2.3% error) for Hive tables [2], and theta sketches for Iceberg tables - Iceberg: Puffin theta sketches [3] - Hive: HLL by default [4] All of them use NDV only for cost-based planning: filter selectivity, join cardinality, and through those, join order and broadcast choice. In conclsion: - Readers don't depend on distinct_count being exact. - Engines that compute their own NDVs choose estimates on purpose. - So I think documenting the field as an estimate matches how it is used today and covers most of the potential future use cases. [1] https://github.com/apache/spark/blob/41f1694f7ba7cc3dd247d00dc42d13f5a8b1eaf7/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala#L399 [2] https://github.com/apache/trino/blob/cf8992f543d0671b2579fded1d17ae3ee862d543/core/trino-main/src/main/java/io/trino/operator/aggregation/DefaultApproximateCountDistinctAggregation.java#L41 [3] https://github.com/apache/iceberg/blob/acee558a84e8dd065bf49fa0af709a5346ede077/format/puffin-spec.md#L114-L123 [4] https://github.com/apache/hive/blob/da789295710cd04b7a76185c6551ec7ef5ab6f1f/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L2798-L2799 On 2026/10/15 20:08:22 Adrian Garcia Badaracco wrote: > I sense the pushback is against saying that writers may use arbitrary > estimates. But unless we agree on how much error is acceptable, or on how > equality is defined, we are in effect allowing an unbounded estimate anyway. > “Approximate” with no bound is the same as “arbitrary”. > > So I think we should pick one of these two options: > > 1. Define the allowed error (or the counting rules) precisely now. > 2. Say explicitly that the value is an estimate with no guaranteed bound. > > I propose option 2 for now. We can come back and add tighter guarantees when > there are concrete use cases that need them, for example a reader that needs > to know the error stays within a specific bound, or that the writer followed > certain rules to compute the distinct count. > Adding a guarantee later is backward compatible because existing readers > already treat the value as a hint. Removing one would not be. > > On 2026/10/07 19:52:38 Daniel Weeks wrote: > > Andrew, I agree with you that the proposed wording is probably not the best. > > > > I don't think the intent was to replace it with arbitrary estimates (though > > the current wording indicates that). > > > > I offered up some alternative language on the PR. > > > > -Dan > > > > > > > > On Wed, Oct 7, 2026 at 12:27 PM Andrew Lamb <[email protected]> wrote: > > > > > IN my mind, the fact that "distinct" is not precisely defined for all > > > types > > > means that changing "count of distinct values occurring" to "Estimated > > > count of distinct values occurring." is a clarification > > > > > > It seems like a significant change in meaning (explicitly allowing > > > estimates) to me <shrug> > > > > > > Andrew > > > > > > On Wed, Oct 7, 2026 at 2:24 PM Daniel Weeks <[email protected]> wrote: > > > > > > > I think the issue is less about whether this is changing something for > > > > older readers/writers or providing a more accurate description. > > > > > > > > Older readers would already "misinterpret" written values because > > > there's a > > > > difference between logical equivalence and physical equivalence. This > > > > difference is left up to interpretation by both the reader and writer. > > > > > > > > An example would be -0 vs +0 for floating point types where some systems > > > > treat them equally where others may distinguish them. Similarly NaN may > > > or > > > > may not be included as a value count. > > > > > > > > I think it's wrong to assume that current readers/writers are capturing > > > > this "correctly" when there is no "correct" or agreedupon > > > > representation. > > > > > > > > -Dan > > > > > > > > On Wed, Oct 7, 2026 at 11:23 AM Russell Spitzer < > > > [email protected] > > > > > > > > > wrote: > > > > > > > > > My understanding is that the exactness of NDV is not standardized at > > > all > > > > > for many types, which makes this value very difficult to define or > > > > > standardize as is. The main callouts during the meeting were strings > > > with > > > > > collations (foo == Foo) floating point (NAN == NAN, -0.0 = 0.0 etc...) > > > So > > > > > since we don't want to spend time addressing these issues, as most use > > > > > cases only require approximate statistics, the consensus was to > > > > > correct > > > > the > > > > > language now rather than spend time establishing something new. Just a > > > > > reminder for everyone, please make sure to add notes when referring to > > > > > discussions that occurred off list. I know the full video and > > > > > trascript > > > > > will eventually be available but it's helpful to keep things straight > > > :) > > > > > From the notes doc - > > > > > > > > > > - > > > > > > > > > > exactness of NDV stats: > > > > > - > > > > > > > > > > Wanted to use in datafusion > > > > > - > > > > > > > > > > Question of exactitude. > > > > > - > > > > > > > > > > Collations makes it difficult because two distinct values might > > > be > > > > > considered equal. > > > > > - > > > > > > > > > > Semantics are under defined. > > > > > - > > > > > > > > > > Do we want to formalize for floating point numbers: -0, +0, > > > > NaNs, > > > > > etc? > > > > > - > > > > > > > > > > Adrian: > > > > > - > > > > > > > > > > Define right now as exact. > > > > > - > > > > > > > > > > Most use cases do not require exact. > > > > > - > > > > > > > > > > Statistical approximations of counts? > > > > > - > > > > > > > > > > Kevin: > > > > > - > > > > > > > > > > do we need to update the doc as well? > > > > > - > > > > > > > > > > Yes! > > > > > > > > > > > > > > > On Wed, Oct 7, 2026 at 1:08 PM Micah Kornfield <[email protected]> > > > > > wrote: > > > > > > > > > > > Hi Adrian, > > > > > > Is there more context from the call, on why Andrew's assessment > > > didn't > > > > > make > > > > > > sense? Are there reference implementations providing approximate > > > > answers > > > > > > today? > > > > > > > > > > > > Thanks, > > > > > > Micah > > > > > > > > > > > > On Wed, Oct 7, 2026 at 10:28 AM Adrian Garcia Badaracco < > > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Per discussion on the sync call today I opened a format PR: > > > > > > > https://github.com/apache/parquet-format/pull/628 > > > > > > > > > > > > > > On 2026/09/30 21:26:04 Andrew Lamb wrote: > > > > > > > > I think optional approximate distinct (or other sketches of the > > > > data > > > > > > like > > > > > > > > histograms) would help various database planners make better > > > plans, > > > > > so > > > > > > > they > > > > > > > > are valuable to explore. > > > > > > > > > > > > > > > > However, I think redefining the existing distinct_count > > > > > > > > statistic > > > > > > > > would have all the forward compatibility problems we have been > > > > > debating > > > > > > > -- > > > > > > > > namely we have no mechanism to prevent older readers > > > > misinterpreting > > > > > a > > > > > > > > field whose semantic meaning was redefined. > > > > > > > > > > > > > > > > So maybe we could investigate adding new standardized metadata > > > > > instead > > > > > > of > > > > > > > > redefining the meaning of an old one > > > > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > On Tue, Sep 22, 2026 at 12:30 PM Adrian Garcia Badaracco via dev > > > < > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > The specification does not define what > > > > `Statistics.distinct_count` > > > > > > > > > contains. Writers do not know if they can write an estimate. > > > > > Readers > > > > > > > do not > > > > > > > > > know if they can use the value as an exact count. > > > > > > > > > > > > > > > > > > The field has this definition in `parquet.thrift`: > > > > > > > > > > > > > > > > > > ```thrift > > > > > > > > > /** count of distinct values occurring */ > > > > > > > > > 4: optional i64 distinct_count; > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > The text does not answer these questions: > > > > > > > > > > > > > > > > > > 1. Must the value be exact? Or can it be an estimate, for > > > example > > > > > > from > > > > > > > a > > > > > > > > > HyperLogLog sketch? > > > > > > > > > > > > > > > > > > If we choose to make the value inexact, this opens up the > > > > question: > > > > > > > > > > > > > > > > > > 1. If the value is not exact, what does it guarantee? Is it an > > > > > upper > > > > > > > > > bound, a lower bound, or an estimate with no bound? > > > > > > > > > > > > > > > > > > If we choose to make the value exact, this opens up other > > > > > questions: > > > > > > > > > > > > > > > > > > 1. How do nulls factor into the count? > > > > > > > > > 2. How do writers compare values? For example, are `-0.0` and > > > > > `+0.0` > > > > > > > one > > > > > > > > > value or two? Are two NaN values with different bit patterns > > > one > > > > > > value > > > > > > > or > > > > > > > > > two? Is the comparison on the physical bytes or on the logical > > > > > type? > > > > > > > > > 3. What does the value count for nested columns? (Relevant: > > > > > > > > > https://github.com/apache/parquet-format/issues/476) > > > > > > > > > > > > > > > > > > It seems to me that an exact count per row group is not very > > > > > useful: > > > > > > > since > > > > > > > > > it cannot be combined across row groups or files any sort of > > > > > > > optimization > > > > > > > > > that attempts to use it for a query like `select > > > > > > > > > count(distinct > > > > > col) > > > > > > > from > > > > > > > > > t` breaks for many queries (also any query with a filter, not > > > to > > > > > > > mention > > > > > > > > > group by, etc.). We could do what Iceberg seems to do and > > > > > > > > > store > > > > > > > sketches > > > > > > > > > that can be combined across files/row groups, but I think that > > > > > should > > > > > > > be a > > > > > > > > > separate discussion as it would involve new fields. > > > > > > > > > > > > > > > > > > Where this does seem like it could be useful is optimizers > > > (e.g. > > > > > join > > > > > > > > > ordering). For those applications I think an inexact value is > > > > just > > > > > as > > > > > > > good > > > > > > > > > as an exact one. > > > > > > > > > > > > > > > > > > Thus I would like to propose that we clarify the spec to mark > > > > this > > > > > > > field > > > > > > > > > as an estimate that can be off in either direction by an > > > > unbounded > > > > > > > amount. > > > > > > > > > If there are good use cases for error estimates we could add > > > that > > > > > > > later, > > > > > > > > > and if we do we can get back exact values by setting the error > > > > to 0 > > > > > > > > > explicitly. The change would be a (possibly) 1 line change in > > > the > > > > > > > comment > > > > > > > > > of the thrift definition above. > > > > > > > > > > > > > > > > > > Below is some investigation I did with an agent to come to > > > > > > > > > this > > > > > > > conclusion. > > > > > > > > > > > > > > > > > > ## History > > > > > > > > > > > > > > > > > > - 2013 (commit `2c4ada8`): Field 4 was `list<ValueStats> > > > > > > > distinct_values`. > > > > > > > > > This was a list of all distinct values and the count of each > > > > value. > > > > > > > > > - 2013 (commit `d9039d0`): The field changed to `i64 > > > > > distinct_count`. > > > > > > > The > > > > > > > > > commit message is "distinct values can be fetched from the > > > > > dictionary > > > > > > > > > page". This shows that the original design was an exact count > > > of > > > > > the > > > > > > > > > dictionary values. But the specification text did not say > > > > > > > > > this. > > > > > > > > > - 2014 (commit `b2836e5`): The comment changed to the current > > > > text. > > > > > > > There > > > > > > > > > is no change in meaning since then. > > > > > > > > > - 2023 (PARQUET-2352, #216): The specification changed > > > > `min_value` > > > > > > and > > > > > > > > > `max_value` from exact values to "lower and upper bound > > > values". > > > > It > > > > > > > also > > > > > > > > > added `is_min_value_exact` and `is_max_value_exact`. This > > > change > > > > > did > > > > > > > not > > > > > > > > > include `distinct_count`. > > > > > > > > > - 2024: A message on [email protected] asked if > > > > > > `distinct_count` > > > > > > > is > > > > > > > > > exact, and if there are use cases for it. The thread discussed > > > > > > > `null_count` > > > > > > > > > only. There was no answer about `distinct_count`. The result > > > was > > > > > > #449, > > > > > > > > > which also changed `null_count` only. Thread: > > > > > > > > > > > > https://lists.apache.org/thread/oqd9lt7p0jtf217hg0667xhk0zvwbgvt > > > > > > > > > - 2014 to now: PARQUET-42 proposes HyperLogLog or Count-Min > > > > > sketches > > > > > > as > > > > > > > > > new statistics. This proposal is not active. > > > > > > > > > > > > > > > > > > ## Current implementations > > > > > > > > > > > > > > > > > > Few implementations write this field. No implementation that > > > > > > > > > we > > > > > > > examined > > > > > > > > > writes an estimate. There are known requests to write > > > estimates, > > > > > > > because an > > > > > > > > > exact count for one row group has limited use for query > > > planning > > > > > (for > > > > > > > > > example, https://github.com/apache/arrow-rs/issues/8608). > > > > > > > > > > > > > > > > > > ### Writers > > > > > > > > > > > > > > > > > > | Implementation | Writes `distinct_count`? | Value | > > > > > > > > > |---|---|---| > > > > > > > > > | parquet-java | No | Not applicable | > > > > > > > > > | Arrow C++ | Only if the user supplies a value | The value > > > from > > > > > the > > > > > > > user | > > > > > > > > > | arrow-go | Only if the user supplies a value | The value > > > > > > > > > from > > > > the > > > > > > > user | > > > > > > > > > | arrow-rs | Optional, off by default (since 2026-08) | > > > Intended > > > > as > > > > > > > exact. > > > > > > > > > It counts 64-bit hashes of the non-null values in the row > > > group. > > > > | > > > > > > > > > | DuckDB | Yes, when the column chunk is dictionary-encoded | > > > The > > > > > > > > > dictionary size | > > > > > > > > > | Impala, cuDF, Polars | No | Not applicable | > > > > > > > > > > > > > > > > > > ### Readers > > > > > > > > > > > > > > > > > > | Implementation | Reads `distinct_count`? | Use | > > > > > > > > > |---|---|---| > > > > > > > > > | parquet-java and engines that use it (Spark, Trino, Presto, > > > > Hive, > > > > > > > Drill) > > > > > > > > > | No | Not applicable | > > > > > > > > > | DuckDB | Yes | Shows the value in `parquet_metadata()` only > > > > > > > > > | > > > > > > > > > | Polars | Yes | Cardinality estimation. It accepts an > > > > approximate > > > > > > > value. | > > > > > > > > > | DataFusion | Yes | Cardinality estimation. If a file has one > > > > row > > > > > > > group, > > > > > > > > > it also uses the value as an exact answer for `COUNT(DISTINCT > > > > > col)`. > > > > > > | > > > > > > > > > > > > > > > > > > ### Comparison: Apache Iceberg > > > > > > > > > > > > > > > > > > Iceberg keeps distinct-value statistics in Puffin files as > > > Apache > > > > > > > > > DataSketches theta sketches. The Puffin specification says > > > > > > > > > that > > > > the > > > > > > > `ndv` > > > > > > > > > property is an "estimate of number of distinct values, derived > > > > from > > > > > > the > > > > > > > > > sketch". Iceberg keeps these estimates separate from its exact > > > > file > > > > > > > > > statistics. > > > > > > > > > > > > > > > > > > ## Possible solutions > > > > > > > > > > > > > > > > > > - Option A (exact): State that distinct_count MUST be the > > > > > > > > > exact > > > > > > number > > > > > > > of > > > > > > > > > distinct values. > > > > > > > > > - Option B (estimate): State that distinct_count is an > > > estimate, > > > > > > which > > > > > > > can > > > > > > > > > be higher or lower than the actual number by an amount that > > > > > > > > > the > > > > > > > > > specification does not limit. > > > > > > > > > - Option C (flag): Add an is_distinct_count_exact flag, > > > > > > > > > similar > > > > to > > > > > > > > > is_min_value_exact and is_max_value_exact. > > > > > > > > > Extension to Options B and C (error bounds): Add optional > > > fields > > > > > for > > > > > > a > > > > > > > > > lower bound, an upper bound and the confidence level of the > > > > bounds. > > > > > > > > > - Option D (separate field): Keep distinct_count exact and add > > > a > > > > > new > > > > > > > field > > > > > > > > > for an estimate or for a sketch that readers can merge, for > > > > example > > > > > > the > > > > > > > > > Apache DataSketches format that Iceberg uses. > > > > > > > > > - Option E (deprecate): Deprecate the field. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
