Hi Dan, I understand the concern about redefining a field. I see this as defining the field for the first time. The field was never defined as accurate, accuracy was simply never specified. No reader expects it to be accurate, and no writer writes completely accurate values.
There is precedent for a similar change for zone map min/max stats: in 2023, PARQUET-2352 (#216 <https://github.com/apache/parquet-format/pull/216>) changed `min_value` and `max_value` from "Min and max values" to "Lower and upper bound values". If you still prefer not to change the text of `distinct_count`, I am fine with the alternative: deprecate `distinct_count` and add a new field. I would propose that the new field be `estimated_distinct_count`, defined as “An estimate of the number of distinct non-null values in the page or column chunk. The estimate can be higher or lower than the actual count by any amount.". This field would be useful for join ordering estimation, which is the use case multiple systems are using the existing field for. It would be easy and performant to compute and use for both readers and writers, and the harm when it is grossly wrong is limited. The main alternative I can think of is a sketch. A sketch can be merged across row groups and files, which helps for single-column joins. But it is larger, more complex to define, and needs agreement on a sketch format. It also does not help much for joins on multiple columns and other more complex cases. So I think starting with a simple per-column per-chunk estimated distinct count integer and adding a sketch later if there are good use cases for it would be the better path forward. Do you see us agreeing on one of these two paths (update the definition to be an estimate or deprecate and add a new field)? Thanks, Adrian On 2026/10/19 14:33:14 Daniel Weeks wrote: > Adrian, I think this issue goes back to Andrew's initial response, which I > agree with: > > However, I think redefining the existing distinct_count statistic > > would have all the forward compatibility problems we have been debating > > > Looking at existing usage and leveraging that to argue changing the meaning > of a field undermines the guarantees we try to provide as a format. > > I can understand clarifying the definition, but this feels like we're > trying to convince ourselves it's ok to redefine the field. > > There are a few scenarios we've discussed where it's reasonable that value > could be ambiguous, but I think generally the expectation should be that it > is accurate. > > If we think there's a better use case for carrying estimated values, maybe > that's good justification for adding a new field (though with the limited > usage you identified, maybe it's not worth the effort). > > -Dan > > On Thu, Oct 15, 2026 at 1:43 PM Adrian Garcia Badaracco <[email protected]> > wrote: > > > To answer Micah's question, I looked at what current implementations do > > with distinct_count, and how engines use NDVs from other sources. > > I am familiar with DataFusion/arrow-rs, but for other systems (DuckDB, > > Polars, Arrow C++, parquet-java, Spark, Trino, Iceberg, Hive and Impala) I > > used an LLM to search their current implementation and did not verify the > > findings by hand. > > Please correct me if I got any of them wrong. > > > > Who writes distinct_count: > > - arrow-rs: opt-in, off by default. It counts a set of value hashes. > > - DuckDB: only for dictionary-encoded chunks (it writes the dictionary > > size). > > - Polars: only for boolean columns. > > - parquet-java, Spark, Trino, Iceberg, Hive and Impala: never. > > > > Who reads it: > > - DataFusion and Polars, and only as a planner estimate. > > - No reader I found uses it for correctness. > > > > Current values are already not exact: > > - Signed zero: arrow-rs and DuckDB both count -0.0 and +0.0 as two values. > > - NaN: arrow-rs compares raw bytes, so NaNs with different bit patterns > > count as different values. > > - Hash collisions: arrow-rs counts xxHash64 hashes, not values, so > > collisions can make the count too low. > > - Dictionary arrays: arrow-rs counts dictionary keys, not values. Arrow > > doesn't require dictionary values to be unique or used, so the count can be > > too high. > > So current writers can be wrong in both directions, and they don't share a > > definition of equality. > > > > Engines with their own NDVs use estimates: > > - Spark: HLL++, 5% default error [1] > > - Trino: approx_distinct (HLL, about 2.3% error) for Hive tables [2], and > > theta sketches for Iceberg tables > > - Iceberg: Puffin theta sketches [3] > > - Hive: HLL by default [4] > > All of them use NDV only for cost-based planning: filter selectivity, join > > cardinality, and through those, join order and broadcast choice. > > > > In conclsion: > > - Readers don't depend on distinct_count being exact. > > - Engines that compute their own NDVs choose estimates on purpose. > > - So I think documenting the field as an estimate matches how it is used > > today and covers most of the potential future use cases. > > > > [1] > > https://github.com/apache/spark/blob/41f1694f7ba7cc3dd247d00dc42d13f5a8b1eaf7/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala#L399 > > [2] > > https://github.com/apache/trino/blob/cf8992f543d0671b2579fded1d17ae3ee862d543/core/trino-main/src/main/java/io/trino/operator/aggregation/DefaultApproximateCountDistinctAggregation.java#L41 > > [3] > > https://github.com/apache/iceberg/blob/acee558a84e8dd065bf49fa0af709a5346ede077/format/puffin-spec.md#L114-L123 > > [4] > > https://github.com/apache/hive/blob/da789295710cd04b7a76185c6551ec7ef5ab6f1f/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L2798-L2799 > > > > On 2026/10/15 20:08:22 Adrian Garcia Badaracco wrote: > > > I sense the pushback is against saying that writers may use arbitrary > > estimates. But unless we agree on how much error is acceptable, or on how > > equality is defined, we are in effect allowing an unbounded estimate anyway. > > > “Approximate” with no bound is the same as “arbitrary”. > > > > > > So I think we should pick one of these two options: > > > > > > 1. Define the allowed error (or the counting rules) precisely now. > > > 2. Say explicitly that the value is an estimate with no guaranteed bound. > > > > > > I propose option 2 for now. We can come back and add tighter guarantees > > when there are concrete use cases that need them, for example a reader that > > needs to know the error stays within a specific bound, or that the writer > > followed certain rules to compute the distinct count. > > > Adding a guarantee later is backward compatible because existing readers > > already treat the value as a hint. Removing one would not be. > > > > > > On 2026/10/07 19:52:38 Daniel Weeks wrote: > > > > Andrew, I agree with you that the proposed wording is probably not the > > best. > > > > > > > > I don't think the intent was to replace it with arbitrary estimates > > (though > > > > the current wording indicates that). > > > > > > > > I offered up some alternative language on the PR. > > > > > > > > -Dan > > > > > > > > > > > > > > > > On Wed, Oct 7, 2026 at 12:27 PM Andrew Lamb <[email protected]> > > wrote: > > > > > > > > > IN my mind, the fact that "distinct" is not precisely defined for > > all types > > > > > means that changing "count of distinct values occurring" to > > "Estimated > > > > > count of distinct values occurring." is a clarification > > > > > > > > > > It seems like a significant change in meaning (explicitly allowing > > > > > estimates) to me <shrug> > > > > > > > > > > Andrew > > > > > > > > > > On Wed, Oct 7, 2026 at 2:24 PM Daniel Weeks <[email protected]> > > wrote: > > > > > > > > > > > I think the issue is less about whether this is changing something > > for > > > > > > older readers/writers or providing a more accurate description. > > > > > > > > > > > > Older readers would already "misinterpret" written values because > > > > > there's a > > > > > > difference between logical equivalence and physical equivalence. > > This > > > > > > difference is left up to interpretation by both the reader and > > writer. > > > > > > > > > > > > An example would be -0 vs +0 for floating point types where some > > systems > > > > > > treat them equally where others may distinguish them. Similarly > > NaN may > > > > > or > > > > > > may not be included as a value count. > > > > > > > > > > > > I think it's wrong to assume that current readers/writers are > > capturing > > > > > > this "correctly" when there is no "correct" or agreedupon > > representation. > > > > > > > > > > > > -Dan > > > > > > > > > > > > On Wed, Oct 7, 2026 at 11:23 AM Russell Spitzer < > > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > My understanding is that the exactness of NDV is not > > standardized at > > > > > all > > > > > > > for many types, which makes this value very difficult to define > > or > > > > > > > standardize as is. The main callouts during the meeting were > > strings > > > > > with > > > > > > > collations (foo == Foo) floating point (NAN == NAN, -0.0 = 0.0 > > etc...) > > > > > So > > > > > > > since we don't want to spend time addressing these issues, as > > most use > > > > > > > cases only require approximate statistics, the consensus was to > > correct > > > > > > the > > > > > > > language now rather than spend time establishing something new. > > Just a > > > > > > > reminder for everyone, please make sure to add notes when > > referring to > > > > > > > discussions that occurred off list. I know the full video and > > trascript > > > > > > > will eventually be available but it's helpful to keep things > > straight > > > > > :) > > > > > > > From the notes doc - > > > > > > > > > > > > > > - > > > > > > > > > > > > > > exactness of NDV stats: > > > > > > > - > > > > > > > > > > > > > > Wanted to use in datafusion > > > > > > > - > > > > > > > > > > > > > > Question of exactitude. > > > > > > > - > > > > > > > > > > > > > > Collations makes it difficult because two distinct values > > might > > > > > be > > > > > > > considered equal. > > > > > > > - > > > > > > > > > > > > > > Semantics are under defined. > > > > > > > - > > > > > > > > > > > > > > Do we want to formalize for floating point numbers: -0, > > +0, > > > > > > NaNs, > > > > > > > etc? > > > > > > > - > > > > > > > > > > > > > > Adrian: > > > > > > > - > > > > > > > > > > > > > > Define right now as exact. > > > > > > > - > > > > > > > > > > > > > > Most use cases do not require exact. > > > > > > > - > > > > > > > > > > > > > > Statistical approximations of counts? > > > > > > > - > > > > > > > > > > > > > > Kevin: > > > > > > > - > > > > > > > > > > > > > > do we need to update the doc as well? > > > > > > > - > > > > > > > > > > > > > > Yes! > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 7, 2026 at 1:08 PM Micah Kornfield < > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Adrian, > > > > > > > > Is there more context from the call, on why Andrew's assessment > > > > > didn't > > > > > > > make > > > > > > > > sense? Are there reference implementations providing > > approximate > > > > > > answers > > > > > > > > today? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Micah > > > > > > > > > > > > > > > > On Wed, Oct 7, 2026 at 10:28 AM Adrian Garcia Badaracco < > > > > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Per discussion on the sync call today I opened a format PR: > > > > > > > > > https://github.com/apache/parquet-format/pull/628 > > > > > > > > > > > > > > > > > > On 2026/09/30 21:26:04 Andrew Lamb wrote: > > > > > > > > > > I think optional approximate distinct (or other sketches > > of the > > > > > > data > > > > > > > > like > > > > > > > > > > histograms) would help various database planners make > > better > > > > > plans, > > > > > > > so > > > > > > > > > they > > > > > > > > > > are valuable to explore. > > > > > > > > > > > > > > > > > > > > However, I think redefining the existing distinct_count > > statistic > > > > > > > > > > would have all the forward compatibility problems we have > > been > > > > > > > debating > > > > > > > > > -- > > > > > > > > > > namely we have no mechanism to prevent older readers > > > > > > misinterpreting > > > > > > > a > > > > > > > > > > field whose semantic meaning was redefined. > > > > > > > > > > > > > > > > > > > > So maybe we could investigate adding new standardized > > metadata > > > > > > > instead > > > > > > > > of > > > > > > > > > > redefining the meaning of an old one > > > > > > > > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > On Tue, Sep 22, 2026 at 12:30 PM Adrian Garcia Badaracco > > via dev > > > > > < > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > The specification does not define what > > > > > > `Statistics.distinct_count` > > > > > > > > > > > contains. Writers do not know if they can write an > > estimate. > > > > > > > Readers > > > > > > > > > do not > > > > > > > > > > > know if they can use the value as an exact count. > > > > > > > > > > > > > > > > > > > > > > The field has this definition in `parquet.thrift`: > > > > > > > > > > > > > > > > > > > > > > ```thrift > > > > > > > > > > > /** count of distinct values occurring */ > > > > > > > > > > > 4: optional i64 distinct_count; > > > > > > > > > > > ``` > > > > > > > > > > > > > > > > > > > > > > The text does not answer these questions: > > > > > > > > > > > > > > > > > > > > > > 1. Must the value be exact? Or can it be an estimate, for > > > > > example > > > > > > > > from > > > > > > > > > a > > > > > > > > > > > HyperLogLog sketch? > > > > > > > > > > > > > > > > > > > > > > If we choose to make the value inexact, this opens up the > > > > > > question: > > > > > > > > > > > > > > > > > > > > > > 1. If the value is not exact, what does it guarantee? Is > > it an > > > > > > > upper > > > > > > > > > > > bound, a lower bound, or an estimate with no bound? > > > > > > > > > > > > > > > > > > > > > > If we choose to make the value exact, this opens up other > > > > > > > questions: > > > > > > > > > > > > > > > > > > > > > > 1. How do nulls factor into the count? > > > > > > > > > > > 2. How do writers compare values? For example, are > > `-0.0` and > > > > > > > `+0.0` > > > > > > > > > one > > > > > > > > > > > value or two? Are two NaN values with different bit > > patterns > > > > > one > > > > > > > > value > > > > > > > > > or > > > > > > > > > > > two? Is the comparison on the physical bytes or on the > > logical > > > > > > > type? > > > > > > > > > > > 3. What does the value count for nested columns? > > (Relevant: > > > > > > > > > > > https://github.com/apache/parquet-format/issues/476) > > > > > > > > > > > > > > > > > > > > > > It seems to me that an exact count per row group is not > > very > > > > > > > useful: > > > > > > > > > since > > > > > > > > > > > it cannot be combined across row groups or files any > > sort of > > > > > > > > > optimization > > > > > > > > > > > that attempts to use it for a query like `select > > count(distinct > > > > > > > col) > > > > > > > > > from > > > > > > > > > > > t` breaks for many queries (also any query with a > > filter, not > > > > > to > > > > > > > > > mention > > > > > > > > > > > group by, etc.). We could do what Iceberg seems to do > > and store > > > > > > > > > sketches > > > > > > > > > > > that can be combined across files/row groups, but I > > think that > > > > > > > should > > > > > > > > > be a > > > > > > > > > > > separate discussion as it would involve new fields. > > > > > > > > > > > > > > > > > > > > > > Where this does seem like it could be useful is > > optimizers > > > > > (e.g. > > > > > > > join > > > > > > > > > > > ordering). For those applications I think an inexact > > value is > > > > > > just > > > > > > > as > > > > > > > > > good > > > > > > > > > > > as an exact one. > > > > > > > > > > > > > > > > > > > > > > Thus I would like to propose that we clarify the spec to > > mark > > > > > > this > > > > > > > > > field > > > > > > > > > > > as an estimate that can be off in either direction by an > > > > > > unbounded > > > > > > > > > amount. > > > > > > > > > > > If there are good use cases for error estimates we could > > add > > > > > that > > > > > > > > > later, > > > > > > > > > > > and if we do we can get back exact values by setting the > > error > > > > > > to 0 > > > > > > > > > > > explicitly. The change would be a (possibly) 1 line > > change in > > > > > the > > > > > > > > > comment > > > > > > > > > > > of the thrift definition above. > > > > > > > > > > > > > > > > > > > > > > Below is some investigation I did with an agent to come > > to this > > > > > > > > > conclusion. > > > > > > > > > > > > > > > > > > > > > > ## History > > > > > > > > > > > > > > > > > > > > > > - 2013 (commit `2c4ada8`): Field 4 was `list<ValueStats> > > > > > > > > > distinct_values`. > > > > > > > > > > > This was a list of all distinct values and the count of > > each > > > > > > value. > > > > > > > > > > > - 2013 (commit `d9039d0`): The field changed to `i64 > > > > > > > distinct_count`. > > > > > > > > > The > > > > > > > > > > > commit message is "distinct values can be fetched from > > the > > > > > > > dictionary > > > > > > > > > > > page". This shows that the original design was an exact > > count > > > > > of > > > > > > > the > > > > > > > [message truncated...]
