Subodh-17 opened a new pull request, #27561:
URL: https://github.com/apache/datafusion/pull/27561
## Which issue does this PR close?
Closes #27456
## Rationale for this change
In Apache DataFusion, partitioned tables can have partitions whose data
files contain zero rows (e.g. created via `COPY (SELECT ... WHERE 1 = 0)` or
generated by upstream ETL pipelines).
Prior to this change, running aggregate queries on a partitioned table
containing an empty partition produced incorrect results when statistics
collection was enabled. For example, for a table with partition `q=1` (1 row)
and partition `q=7` (0 rows):
```sql
SELECT count(*), min(q), max(q) FROM t;
```
returned `(1, 1, 7)` with statistics enabled, but `(1, 1, 1)` when
statistics were disabled.
### Root Cause
`PartitionedFile::with_statistics` unconditionally synthesized exact column
statistics:
```rust
(
Precision::Exact(0),
Precision::Exact(partition_value.clone()),
Precision::Exact(1),
)
```
for partition columns, regardless of whether the file contained rows. As a
result, an empty file with `stats.num_rows == Precision::Exact(0)` declared an
exact minimum and maximum equal to its partition value (e.g. `min=7, max=7`).
When `Statistics::try_merge_iter_with_ndv_fallback` merged file statistics
into table-level statistics, these spurious extrema were included in the
table's bounds. The `AggregateStatistics` physical optimizer then substituted
constant literals `min(q) -> 1` and `max(q) -> 7` into the plan without
scanning physical data, returning corrupt aggregate results.
## What changes are included in this PR?
1. **`datafusion/datasource/src/mod.rs`**:
- In `PartitionedFile::with_statistics`, added an explicit check for
`stats.num_rows == Precision::Exact(0)`:
```rust
let (null_count, min_max, distinct_count) =
if stats.num_rows == Precision::Exact(0) {
(Precision::Exact(0), Precision::Absent, Precision::Exact(0))
} else if partition_value.is_null() {
(stats.num_rows, Precision::Absent, Precision::Exact(0))
} else {
(
Precision::Exact(0),
Precision::Exact(partition_value.clone()),
Precision::Exact(1),
)
};
```
- An empty file contains 0 rows; therefore, its partition column contains
0 distinct values, 0 nulls, and its min/max extrema are absent
($\min(\emptyset) = \text{undefined}$).
- `distinct_count` is assigned `Precision::Exact(0)` rather than
`Precision::Absent`. This ensures that multi-file NDV merging in
`Statistics::try_merge_iter_with_ndv_fallback` matches `(Some(&l), Some(&0))`
and preserves the known table NDV via `ndv_fallback.merge(l, 0) = max(l, 0) =
l`, preventing unnecessary degradation to `Precision::Absent`.
- Updated docstrings on `struct PartitionedFile` and `fn with_statistics`
documenting the empty-file statistics invariant.
- Added unit tests:
- `test_with_statistics_empty_file_non_null_partition`: Verifies an
empty file with a non-null partition value (`q = 7`) yields `null_count =
Exact(0)`, `min = Absent`, `max = Absent`, `distinct_count = Exact(0)`, and
`byte_size = Exact(0)`.
- `test_with_statistics_empty_file_null_partition`: Verifies an empty
file with a Hive null partition value (`q = NULL`) yields identical empty
statistics.
2. **`datafusion/sqllogictest/test_files/listing_table_partitions.slt`**:
- Added end-to-end sqllogictest reproducing the issue scenario:
- Partition `q=1` with 1 row.
- Partition `q=7` with 0 rows.
- Verifies `SELECT count(*), min(q), max(q)` returns `1 1 1`.
- Verifies `GROUP BY q` returns only active partitions (`1 1 1 1`).
- Verifies parity when `set datafusion.execution.collect_statistics =
false`.
## Are these changes tested?
Yes:
- **Targeted unit tests:**
```bash
cargo test -p datafusion-datasource -- test_with_statistics
```
Result: 4 passed, 0 failed.
- **Full crate test suite:**
```bash
cargo test -p datafusion-datasource
```
Result: 205 passed, 0 failed; 8 doctests passed.
- **Sqllogictests:**
```bash
cargo test -p datafusion-sqllogictest --test sqllogictests --
listing_table_partitions
```
Result: 1/1 files completed (100%), 0 failures.
- **Code formatting:**
```bash
cargo fmt --check
```
Result: Clean.
- **Clippy:**
```bash
cargo clippy -p datafusion-datasource -- -D warnings
```
Result: Clean (0 warnings).
## Are there any user-facing changes?
Yes: Queries evaluating `MIN` or `MAX` aggregates over partitioned tables
with empty partition files will no longer return incorrect partition extrema
derived from the empty files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]