adriangb opened a new issue, #24109:
URL: https://github.com/apache/datafusion/issues/24109
### Describe the bug
With `datafusion.execution.parquet.pushdown_filters = true`, a predicate on
a struct field is reported as fully handled by the scan — so `FilterExec` is
removed from the plan — and then never applied. The query returns **every row**.
The trigger is not `get_field` itself; it is the expression adapter
interposing a cast on the struct column, which happens whenever the table's
declared schema differs from the file's physical schema for that column.
### To Reproduce
```sql
-- file written with: id Int64, s Struct<x Int64, y Utf8>, v Int64, three
rows, x = 100/200/300
CREATE EXTERNAL TABLE fullt (id INT, s STRUCT<x INT, y VARCHAR>, v INT)
STORED AS PARQUET LOCATION '.../w.parquet';
set datafusion.execution.parquet.pushdown_filters = true;
SELECT id FROM fullt WHERE v = 20; -- correct: 1 row
SELECT id FROM fullt WHERE s['x'] = 200; -- WRONG: 3 rows
```
The same query against a table whose schema is *inferred* from the file — so
no cast is inserted — filters correctly. Here the declared `INT` differs from
the file's `Int64`, and `VARCHAR` maps to `Utf8View` against the file's `Utf8`,
so a cast is inserted.
`EXPLAIN` for the broken case shows no `FilterExec` anywhere; the scan has
claimed the predicate:
```
logical_plan
01)Projection: fullt.id
02)--Filter: __datafusion_extracted_1 = Int32(200)
03)----Projection: get_field(fullt.s, Utf8("x")) AS
__datafusion_extracted_1, fullt.id
04)------TableScan: fullt projection=[id, s],
partial_filters=[get_field(fullt.s, Utf8("x")) = Int32(200)]
physical_plan DataSourceExec: ... projection=[id, s],
predicate=get_field(s@1, x) = 200
```
### Expected behavior
One row. A predicate the scan reports as exactly handled must actually be
applied.
### Mechanism
1. `ParquetSource::try_pushdown_filters`
(`datasource-parquet/src/source.rs`) decides exactness with
`can_expr_be_pushed_down_with_schemas(&filter, pushable_schema)`, where
`pushable_schema` is the **table** schema. Against it the predicate is
`get_field(Column("s"), 'x') = 200`, which `PushdownChecker` accepts, so the
filter is marked `PushedDown::Yes` and `FilterExec` is removed.
2. At file-open time the physical expression adapter rewrites the predicate
against the **physical file** schema, producing `get_field(CAST(s AS Struct{x:
Int32, y: Utf8View}), 'x') = 200`.
3. `PushdownChecker::f_down` only recognises `get_field` whose first
argument is a bare `Column`. Here it is a `CastExpr`, so the visitor descends
past it, reaches `Column("s")`, sees a `Struct`, sets `non_primitive_columns =
true`, and `pushdown_columns` returns `None`.
4. `build_row_filter` collects no candidate for that conjunct and returns
`Ok(None)`. The conjunct is dropped, and nothing downstream re-applies it —
step 1 already removed `FilterExec`.
### This is an instance of a general category
The specific `get_field`-over-a-cast case is easy enough to patch, but the
shape of the bug will recur: **a static determination made at planning time
about what the scan can do, and the runtime construction that has to make good
on it, are computed by different code against different schemas, and there is
no mechanism forcing them to agree.** Planning decides against the table
schema; the row filter is built per file against that file's physical schema
after adaptation. Any divergence between the two turns a "predicate handled
exactly" promise into a silently dropped predicate.
The codebase already knows this hazard exists and patches it case by case —
`validate_predicate_does_not_reference_virtual_columns` in `opener/mod.rs`
exists for exactly this reason, and its doc comment says "Silently dropping
such a predicate would produce wrong results." That is one instance guarded by
hand; this is another; there are likely more, and each new pushdown-eligible
expression shape is a chance to add one.
The structural fix is to remove the cliff: **let `ParquetOpener` apply a
predicate as an ordinary post-decode filter when it cannot be turned into a row
filter.** Then failing to build a row filter — for any reason, known or not yet
discovered — costs at most some performance instead of correctness, and the
planner's "exactly handled" claim stays true regardless of what the per-file
construction manages to do. Individual cases can then be optimised into row
filters as they are understood, rather than each unhandled case being a latent
wrong-results bug.
### Additional context
Found while reviewing apache/datafusion#24090 (nested projection pruning for
Parquet). That PR does not cause this — `row_filter.rs` is untouched by it and
the failing query takes an unchanged code path — but it is worth flagging
alongside it, because a table whose declared nested schema is narrower than the
file's *always* carries the adapter cast that triggers this. So on exactly the
tables that feature targets, any `WHERE s['field'] = ...` with
`pushdown_filters = true` is wrong today.
It is also why nested pruning cannot simply be extended to the filter path
as-is: filter pushdown over a narrowed nested column is currently unsound, not
merely unoptimised. Teaching `PushdownChecker` to look through the cast would
address both, but only if the fallback above exists to catch whatever it still
misses.
cc @alamb
---
_Generated by [Claude Code](https://claude.ai/code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
