txwyy123 commented on PR #27189:
URL: https://github.com/apache/datafusion/pull/27189#issuecomment-7028026893

   Thanks @alamb -- good point. From an end user perspective:
   
   Since #26310, DataFusion reads Hive `__HIVE_DEFAULT_PARTITION__` directories 
as NULL partition values. A side effect is that a single NULL partition makes 
the table-level min/max statistics for that partition column `Absent`, so 
queries that could be answered from metadata now scan every file.
   
   Example: a table partitioned by `d` with `d=2024-01-01/`, `d=2024-12-31/` 
and `d=__HIVE_DEFAULT_PARTITION__/`:
   
   ```sql
   EXPLAIN SELECT min(d), max(d) FROM t;
   ```
   
   On main, this scans all files:
   ```
   AggregateExec: mode=Final ...
     CoalescePartitionsExec
       AggregateExec: mode=Partial ...
         DataSourceExec: file_groups={3 groups: [...]}
   ```
   
   With this PR, it is answered from statistics with no I/O:
   ```
   ProjectionExec: expr=[2024-01-01 as min(t.d), 2024-12-31 as max(t.d)]
     PlaceholderRowExec
   ```
   
   The same `Absent` bounds also degrade filter selectivity estimates and other 
stats-based decisions. Results are correct either way; this is a performance 
regression for tables with NULL partitions.
   
   I'll update the PR description to lead with this, and change the 
sqllogictest to assert on the `EXPLAIN` plan so it fails without the fix.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to