Rich-T-kid opened a new issue, #10650: URL: https://github.com/apache/arrow-rs/issues/10650
### Is your feature request related to a problem or challenge? There are several open DataFusion issues ((https://github.com/apache/datafusion/issues/24111), (https://github.com/apache/datafusion/issues/24114), (https://github.com/apache/datafusion/issues/22891)) around using cardinality information to drive better query optimization -- things like choosing dictionary encoding for low-cardinality string columns, improving join ordering, and producing more accurate NDV estimates for aggregations. All of these depend on the optimizer having access to distinct value counts per column at planning time. In DataFusion, this information flows in through `TableProvider::statistics()`. When a parquet-backed table is scanned, DataFusion calls through `ParquetFormat::infer_stats_and_ordering()` into `statistics_from_parquet_metadata()` in `datasource-parquet/src/metadata.rs`, which uses arrow-rs's StatisticsConverter to read per-column statistics out of the parquet file footer and build ColumnStatistics. This all happens before any data pages are read, it's purely metadata. The problem is that StatisticsConverter exposes `row_group_mins`, `row_group_maxes`, and `row_group_null_counts`, but has no equivalent for `distinct_count`. So `ColumnStatistics::distinct_count` is always `Precision::Absent` for parquet files, regardless of whether the file actually contains that information. The optimizer never sees cardinality data for any parquet column, and the DataFusion issues above are blocked on it. arrow-rs (currently) doesn't write this meta data but other writers may and we aren't taking advantage of it. ### Describe the solution you'd like **Read side** : targeting arrow-rs 60 The most impactful and lowest-risk fix is on the read path. StatisticsConverter in `parquet/src/arrow/arrow_reader/statistics.rs` needs a `row_group_distinct_counts()` method that follows the same pattern as r`ow_group_null_counts()`, reading `distinct_count_opt()` from each row group's column chunk metadata and returning a UInt64Array with nulls where the value is absent. This is the right place to start because parquet files written by other implementations, I.E parquet-java, parquet-cpp, may already have `distinct_count` populated. Wiring up the read path gives DataFusion immediate value from those files with no write-side changes required and no behavioral restrictions. Once the method exists in arrow-rs, the DataFusion change is straightforward: `summarize_column_statistics()` in `datasource-parquet/src/metadata.rs` calls the new method and uses it to populate `ColumnStatistics::distinct_count` as` Precision::Exact` when all row groups have the value, or `Precision::Inexact` when it's only partially available. **Write side** The ArrowWriter currently never populates `distinct_count` in the statistics it writes. The field exists (column_distinct_count: Option<u64> in ColumnMetrics in parquet/src/column/writer/mod.rs) but only the low-level write_batch_with_statistics() API can reach it — the standard ArrowWriter path never fills it in. Doing this accurately requires some form of counting per column per row group. A HashSet<T> gives exact values but memory usage grows with cardinality, which gets expensive for high-cardinality string columns across large row groups. A HyperLogLog gives approximate counts with bounded memory but introduces estimation error that would need to surface as `Precision::Inexact`. The right tradeoff isn't obvious and we'd want to benchmark write overhead before settling on an approach. This could also be opt-in through WriterProperties rather than on by default, since not every parquet writer needs it and the overhead isn't free. The write side is worth doing but shouldn't hold up the read path, which is the cleaner immediate win with no restrictions. ### Describe alternatives you've considered n/a ### Additional context https://github.com/apache/datafusion/issues/24111 https://github.com/apache/datafusion/issues/24114 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
