fordN opened a new pull request, #23395:
URL: https://github.com/apache/datafusion/pull/23395
## Which issue does this PR close?
- Closes #23220.
## Rationale for this change
The reported bug showed that when volatile (non-deterministic) expressions
are referenced multiple times in an outer function the query results can be
incorrect.
To produce correct results, a volatile expression (e.g. `random()`,
`uuid()`) aliased once in a subquery and referenced multiple times must be
evaluated once and reused. Since 52.0.0 this evaluation pattern for volatile
functions has been broken; the physical projection-pushdown rule merges the
outer projection into the file `DataSourceExec`, inlining the aliased volatile
expression at each reference site. Instead of being evaluated once, the
volatile expression is evaluated N times, so the references diverge:
```sql
SELECT s.r AS x, s.r AS y FROM (SELECT random() AS r FROM t) AS s; -- x !=
y on >= 52.0.0
```
This was correct in 51.0.0 and regressed in 52.0.0/53.0.0. It reproduces on
file scans (Parquet/CSV) but not in-memory tables, and was surfaced downstream
in Ibis. Worth noting, #10337 appears to report the same class of bug but a
different cause (I'll try to look into that issue soon as well).
## What changes are included in this PR?
`FileScanConfig::try_swapping_with_projection` now declines to merge a
projection into the file source when the merge would inline a volatile
expression that the incoming projection references. The check reuses the
existing `is_volatile()` utility from `datafusion_physical_expr_common` — the
same volatility gate the physical `ProjectionPushdown` and `FilterPushdown`
rules already apply — so file-source projection merging is now consistent with
them. Deterministic expressions still merge freely.
The guard blocks when a volatile inner expression is referenced at least
once (not only more than once), because a single outer expression can itself
duplicate the value (e.g.`r + r`).
## Are these changes tested?
Yes:
- Unit tests in `file_scan_config.rs`:
- volatile referenced ≥1× → blocked;
- deterministic computed and column-only → allowed;
- unreferenced volatile → allowed;
- single-expression self-reference (`r + r`) → blocked;
- volatile nested in arithmetic → blocked;
- empty → allowed.
- A regression test in `projection_pushdown.slt` asserting `x = y` for the
aliased-`random()` pattern over a Parquet scan.
- Full `datafusion-sqllogictest` suite passes.
## Are there any user-facing changes?
No API changes! Query results for the affected pattern are corrected (a
volatile expression aliased in a subquery is no longer duplicated by projection
pushdown into a file scan). Physical plans for such queries now retain a
`ProjectionExec` above the scan rather than inlining the volatile expression
into the `DataSourceExec` projection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
