txwyy123 commented on PR #27189:
URL: https://github.com/apache/datafusion/pull/27189#issuecomment-7028026893
Thanks @alamb -- good point. From an end user perspective:
Since #26310, DataFusion reads Hive `__HIVE_DEFAULT_PARTITION__` directories
as NULL partition values. A side effect is that a single NULL partition makes
the table-level min/max statistics for that partition column `Absent`, so
queries that could be answered from metadata now scan every file.
Example: a table partitioned by `d` with `d=2024-01-01/`, `d=2024-12-31/`
and `d=__HIVE_DEFAULT_PARTITION__/`:
```sql
EXPLAIN SELECT min(d), max(d) FROM t;
```
On main, this scans all files:
```
AggregateExec: mode=Final ...
CoalescePartitionsExec
AggregateExec: mode=Partial ...
DataSourceExec: file_groups={3 groups: [...]}
```
With this PR, it is answered from statistics with no I/O:
```
ProjectionExec: expr=[2024-01-01 as min(t.d), 2024-12-31 as max(t.d)]
PlaceholderRowExec
```
The same `Absent` bounds also degrade filter selectivity estimates and other
stats-based decisions. Results are correct either way; this is a performance
regression for tables with NULL partitions.
I'll update the PR description to lead with this, and change the
sqllogictest to assert on the `EXPLAIN` plan so it fails without the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]