The specification does not define what `Statistics.distinct_count` contains. Writers do not know if they can write an estimate. Readers do not know if they can use the value as an exact count.
The field has this definition in `parquet.thrift`: ```thrift /** count of distinct values occurring */ 4: optional i64 distinct_count; ``` The text does not answer these questions: 1. Must the value be exact? Or can it be an estimate, for example from a HyperLogLog sketch? If we choose to make the value inexact, this opens up the question: 1. If the value is not exact, what does it guarantee? Is it an upper bound, a lower bound, or an estimate with no bound? If we choose to make the value exact, this opens up other questions: 1. How do nulls factor into the count? 2. How do writers compare values? For example, are `-0.0` and `+0.0` one value or two? Are two NaN values with different bit patterns one value or two? Is the comparison on the physical bytes or on the logical type? 3. What does the value count for nested columns? (Relevant: https://github.com/apache/parquet-format/issues/476) It seems to me that an exact count per row group is not very useful: since it cannot be combined across row groups or files any sort of optimization that attempts to use it for a query like `select count(distinct col) from t` breaks for many queries (also any query with a filter, not to mention group by, etc.). We could do what Iceberg seems to do and store sketches that can be combined across files/row groups, but I think that should be a separate discussion as it would involve new fields. Where this does seem like it could be useful is optimizers (e.g. join ordering). For those applications I think an inexact value is just as good as an exact one. Thus I would like to propose that we clarify the spec to mark this field as an estimate that can be off in either direction by an unbounded amount. If there are good use cases for error estimates we could add that later, and if we do we can get back exact values by setting the error to 0 explicitly. The change would be a (possibly) 1 line change in the comment of the thrift definition above. Below is some investigation I did with an agent to come to this conclusion. ## History - 2013 (commit `2c4ada8`): Field 4 was `list<ValueStats> distinct_values`. This was a list of all distinct values and the count of each value. - 2013 (commit `d9039d0`): The field changed to `i64 distinct_count`. The commit message is "distinct values can be fetched from the dictionary page". This shows that the original design was an exact count of the dictionary values. But the specification text did not say this. - 2014 (commit `b2836e5`): The comment changed to the current text. There is no change in meaning since then. - 2023 (PARQUET-2352, #216): The specification changed `min_value` and `max_value` from exact values to "lower and upper bound values". It also added `is_min_value_exact` and `is_max_value_exact`. This change did not include `distinct_count`. - 2024: A message on [email protected] asked if `distinct_count` is exact, and if there are use cases for it. The thread discussed `null_count` only. There was no answer about `distinct_count`. The result was #449, which also changed `null_count` only. Thread: https://lists.apache.org/thread/oqd9lt7p0jtf217hg0667xhk0zvwbgvt - 2014 to now: PARQUET-42 proposes HyperLogLog or Count-Min sketches as new statistics. This proposal is not active. ## Current implementations Few implementations write this field. No implementation that we examined writes an estimate. There are known requests to write estimates, because an exact count for one row group has limited use for query planning (for example, https://github.com/apache/arrow-rs/issues/8608). ### Writers | Implementation | Writes `distinct_count`? | Value | |---|---|---| | parquet-java | No | Not applicable | | Arrow C++ | Only if the user supplies a value | The value from the user | | arrow-go | Only if the user supplies a value | The value from the user | | arrow-rs | Optional, off by default (since 2026-08) | Intended as exact. It counts 64-bit hashes of the non-null values in the row group. | | DuckDB | Yes, when the column chunk is dictionary-encoded | The dictionary size | | Impala, cuDF, Polars | No | Not applicable | ### Readers | Implementation | Reads `distinct_count`? | Use | |---|---|---| | parquet-java and engines that use it (Spark, Trino, Presto, Hive, Drill) | No | Not applicable | | DuckDB | Yes | Shows the value in `parquet_metadata()` only | | Polars | Yes | Cardinality estimation. It accepts an approximate value. | | DataFusion | Yes | Cardinality estimation. If a file has one row group, it also uses the value as an exact answer for `COUNT(DISTINCT col)`. | ### Comparison: Apache Iceberg Iceberg keeps distinct-value statistics in Puffin files as Apache DataSketches theta sketches. The Puffin specification says that the `ndv` property is an "estimate of number of distinct values, derived from the sketch". Iceberg keeps these estimates separate from its exact file statistics. ## Possible solutions - Option A (exact): State that distinct_count MUST be the exact number of distinct values. - Option B (estimate): State that distinct_count is an estimate, which can be higher or lower than the actual number by an amount that the specification does not limit. - Option C (flag): Add an is_distinct_count_exact flag, similar to is_min_value_exact and is_max_value_exact. Extension to Options B and C (error bounds): Add optional fields for a lower bound, an upper bound and the confidence level of the bounds. - Option D (separate field): Keep distinct_count exact and add a new field for an estimate or for a sketch that readers can merge, for example the Apache DataSketches format that Iceberg uses. - Option E (deprecate): Deprecate the field.
