hcrosse opened a new issue, #27622:
URL: https://github.com/apache/datafusion/issues/27622
### Describe the bug
I get no rows instead of two matching `NaN` rows after disabling Parquet
pruning before creating the external table.
When `datafusion.execution.parquet.pruning` is set to `false` before
creating a Parquet external table (and `enable_page_index` and
`bloom_filter_on_read` are also `false`), `ListingTable` still prunes the
Parquet file using its collected statistics during planning and replaces the
scan with `EmptyExec`.
This makes `datafusion.execution.parquet.pruning = false` unable to disable
Parquet statistics pruning on a `ListingTable` (for example as a workaround for
#15812 or files with untrusted statistics). Disabling
`datafusion.execution.collect_statistics` also bypasses this pruning.
### To Reproduce
I reproduced this on `main` (`57.0.0-dev`):
```sql
COPY (
SELECT column1 AS id, column2 AS f
FROM (VALUES (1, CAST('NaN' AS DOUBLE)), (2, CAST('NaN' AS DOUBLE)), (3,
0.0::DOUBLE), (4, 1.0::DOUBLE))
) TO 'nan_rg.parquet';
SET datafusion.execution.parquet.pruning = false;
SET datafusion.execution.parquet.enable_page_index = false;
SET datafusion.execution.parquet.bloom_filter_on_read = false;
CREATE EXTERNAL TABLE t STORED AS PARQUET LOCATION 'nan_rg.parquet';
SELECT id, f FROM t WHERE f = CAST('NaN' AS DOUBLE);
```
Observed result: `0` rows.
`EXPLAIN SELECT id, f FROM t WHERE f = CAST('NaN' AS DOUBLE)` shows the file
was pruned before building the Parquet scan:
```text
logical_plan:
Filter: t.f = Float64(NaN)
TableScan: t projection=[id, f], partial_filters=[t.f = Float64(NaN)]
physical_plan:
FilterExec: f@1 = NaN
EmptyExec
```
### Expected behavior
With `datafusion.execution.parquet.pruning = false` (along with
`enable_page_index = false` and `bloom_filter_on_read = false`), the Parquet
file should not be pruned using its Parquet min and max statistics, and the
query should return the 2 matching `NaN` rows `(1, NaN)` and `(2, NaN)`.
### Additional context
In #26441, `ListingTable::file_statistics_pruning_predicate` checks only
`ctx.config().collect_statistics()` and `self.expr_adapter_factory.is_some()`:
https://github.com/apache/datafusion/blob/904afffc731637c4c927c8e095ed07fb195a6c79/datafusion/catalog-listing/src/table.rs#L1029-L1053
This path does not check `ParquetFormat::enable_pruning()`, which returns
`options.global.pruning`.
I would be happy to contribute a fix once maintainers confirm the preferred
direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]