alamb opened a new issue, #23531:
URL: https://github.com/apache/datafusion/issues/23531
### Describe the bug
`input_file_name()` is a source-dependent function that is evaluated by the
file scan, not from column data, so it cannot be evaluated inside a Parquet
`RowFilter`. When `datafusion.execution.parquet.pushdown_filters` is enabled, a
predicate that references `input_file_name()` gets pushed down into the Parquet
scan, and the query fails at execution time with:
```
Parquet error: External: Compute error: Error evaluating filter predicate:
input_file_name() is source dependent and cannot be evaluated directly
```
The sibling source-dependent function `file_row_index()` does *not* have
this problem because it is explicitly excluded from filter pushdown in
`PushdownChecker`
(`datafusion/datasource-parquet/src/projection_read_plan.rs`).
`input_file_name()` (`InputFileNameFunc`, added in #22978) is not excluded.
### To Reproduce
Run in `datafusion-cli` (needs a writable `/tmp`):
```sql
COPY (VALUES (10), (20), (30))
TO '/tmp/df_ifn_repro/first.parquet'
STORED AS PARQUET;
CREATE EXTERNAL TABLE t(column1 INT)
STORED AS PARQUET
LOCATION '/tmp/df_ifn_repro/';
-- pushdown disabled (default): works, returns 10, 20, 30
SELECT column1 FROM t
WHERE input_file_name() LIKE '%first.parquet'
ORDER BY column1;
-- enable parquet filter pushdown
SET datafusion.execution.parquet.pushdown_filters = true;
-- BUG: same query now errors instead of returning 10, 20, 30
SELECT column1 FROM t
WHERE input_file_name() LIKE '%first.parquet'
ORDER BY column1;
```
The first `SELECT` returns `10, 20, 30`; the second errors.
### Expected behavior
The query should return `10, 20, 30` regardless of the `pushdown_filters`
setting. Predicates referencing source-dependent functions like
`input_file_name()` should not be pushed into the Parquet `RowFilter`; they
should stay in a `FilterExec` above the scan (as already happens for
`file_row_index()`).
### Additional context
- The `input_file_name()` UDF was added in #22978.
- Likely fix: exclude `InputFileNameFunc` (and, more generally,
source-dependent functions) from pushdown in `PushdownChecker`, mirroring the
existing `FileRowIndexFunc` handling.
- This currently blocks enabling `parquet.pushdown_filters` by default
(#3463): with the default flipped, this query errors instead of just being
slower.
- A failing regression test (added to `input_file_name.slt`) is on branch
[`qizhu/parquet-pushdown-source-dependent-filter-bug`](https://github.com/alamb/datafusion/tree/qizhu/parquet-pushdown-source-dependent-filter-bug).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
