adriangb commented on issue #19092:
URL: https://github.com/apache/datafusion/issues/19092#issuecomment-3679172251
Here's the reproducer for that:
```sql
CREATE EXTERNAL TABLE t (
x INT
)
STORED AS PARQUET
LOCATION '/tmp/test-data/';
INSERT INTO t VALUES (1), (1);
EXPLAIN ANALYZE
SELECT COUNT(*) FROM t WHERE x IS NULL;
```
As you can see if has `files_ranges_pruned_statistics=1 total → 1 matched`.
Adding a simplifier for `FilePruner` I see `files_ranges_pruned_statistics=1
total → 0 matched`.
The diff is:
```diff
diff --git a/datafusion/datasource-parquet/src/opener.rs
b/datafusion/datasource-parquet/src/opener.rs
index bea970f14..2f1dd192f 100644
--- a/datafusion/datasource-parquet/src/opener.rs
+++ b/datafusion/datasource-parquet/src/opener.rs
@@ -303,9 +303,17 @@ impl FileOpener for ParquetOpener {
// 10` could hit this condition.
is_dynamic_physical_expr(p) ||
partitioned_file.has_statistics()
})
- .and_then(|p| {
+ .cloned()
+ .and_then(|mut p| {
+ // If we made any replacements of columns with literals
run a simplifier pass.
+ // This handles cases like `literal_col is not null` ->
`true`.
+ if !literal_columns.is_empty() {
+ let simplifier =
+
PhysicalExprSimplifier::new(&logical_file_schema);
+ p = simplifier.simplify(p).ok()?;
+ }
FilePruner::try_new(
- Arc::clone(p),
+ p,
&logical_file_schema,
&partitioned_file,
predicate_creation_errors.clone(),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
