hcrosse opened a new issue, #27622:
URL: https://github.com/apache/datafusion/issues/27622

   ### Describe the bug
   
   I get no rows instead of two matching `NaN` rows after disabling Parquet 
pruning before creating the external table.
   
   When `datafusion.execution.parquet.pruning` is set to `false` before 
creating a Parquet external table (and `enable_page_index` and 
`bloom_filter_on_read` are also `false`), `ListingTable` still prunes the 
Parquet file using its collected statistics during planning and replaces the 
scan with `EmptyExec`.
   
   This makes `datafusion.execution.parquet.pruning = false` unable to disable 
Parquet statistics pruning on a `ListingTable` (for example as a workaround for 
#15812 or files with untrusted statistics). Disabling 
`datafusion.execution.collect_statistics` also bypasses this pruning.
   
   ### To Reproduce
   
   I reproduced this on `main` (`57.0.0-dev`):
   
   ```sql
   COPY (
     SELECT column1 AS id, column2 AS f
     FROM (VALUES (1, CAST('NaN' AS DOUBLE)), (2, CAST('NaN' AS DOUBLE)), (3, 
0.0::DOUBLE), (4, 1.0::DOUBLE))
   ) TO 'nan_rg.parquet';
   
   SET datafusion.execution.parquet.pruning = false;
   SET datafusion.execution.parquet.enable_page_index = false;
   SET datafusion.execution.parquet.bloom_filter_on_read = false;
   
   CREATE EXTERNAL TABLE t STORED AS PARQUET LOCATION 'nan_rg.parquet';
   
   SELECT id, f FROM t WHERE f = CAST('NaN' AS DOUBLE);
   ```
   
   Observed result: `0` rows.
   
   `EXPLAIN SELECT id, f FROM t WHERE f = CAST('NaN' AS DOUBLE)` shows the file 
was pruned before building the Parquet scan:
   
   ```text
   logical_plan:
     Filter: t.f = Float64(NaN)
       TableScan: t projection=[id, f], partial_filters=[t.f = Float64(NaN)]
   physical_plan:
     FilterExec: f@1 = NaN
       EmptyExec
   ```
   
   ### Expected behavior
   
   With `datafusion.execution.parquet.pruning = false` (along with 
`enable_page_index = false` and `bloom_filter_on_read = false`), the Parquet 
file should not be pruned using its Parquet min and max statistics, and the 
query should return the 2 matching `NaN` rows `(1, NaN)` and `(2, NaN)`.
   
   ### Additional context
   
   In #26441, `ListingTable::file_statistics_pruning_predicate` checks only 
`ctx.config().collect_statistics()` and `self.expr_adapter_factory.is_some()`:
   
   
https://github.com/apache/datafusion/blob/904afffc731637c4c927c8e095ed07fb195a6c79/datafusion/catalog-listing/src/table.rs#L1029-L1053
   
   This path does not check `ParquetFormat::enable_pruning()`, which returns 
`options.global.pruning`.
   
   I would be happy to contribute a fix once maintainers confirm the preferred 
direction.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to