galaxytemple opened a new issue, #27418:
URL: https://github.com/apache/datafusion/issues/27418
### Describe the bug
When a projection outputs the same volatile expression twice (e.g.
`floor(random() * 3) AS x, floor(random() * 3) AS y`),
`EquivalenceGroup::project` treats them as equal (`x = y`) because both targets
share the same source expression key.
Since `x` and `y` evaluate independently to different random values at
runtime, this false `x = y` equivalence causes `ORDER BY x, y` to drop `y` from
sorting (`SortExec: expr=[x@0 ASC NULLS LAST]`), returning rows where `y` is
unsorted within equal values of `x`.
Furthermore, in a `UNION ALL`, both arms carry this false `x = y`, causing
`UnionExec` to preserve `x = y` across the union and break downstream ordering.
### To Reproduce
In `datafusion-cli`:
### Case 1: Single Projection (Direct Out-of-Order Result)
```sql
SELECT x, y FROM (
SELECT floor(random() * 3) AS x, floor(random() * 3) AS y FROM
generate_series(1, 10)
) ORDER BY x, y;
```
Output (`y` is unsorted within `x = 0.0` or `x = 2.0`):
```text
+-----+-----+
| x | y |
+-----+-----+
| 0.0 | 2.0 |
| 0.0 | 1.0 |
| 0.0 | 0.0 |
| 0.0 | 2.0 |
| 1.0 | 1.0 |
| 2.0 | 0.0 |
| 2.0 | 1.0 |
| 2.0 | 2.0 |
| 2.0 | 0.0 |
| 2.0 | 0.0 |
+-----+-----+
```
`EXPLAIN` shows `SortExec` sorting by `x` only because it believes `x = y`:
```text
SortPreservingMergeExec: [x@0 ASC NULLS LAST, y@1 ASC NULLS LAST]
SortExec: expr=[x@0 ASC NULLS LAST], preserve_partitioning=[true]
ProjectionExec: expr=[floor(random() * 3) as x, floor(random() * 3) as y]
```
### Case 2: UNION ALL (Propagates False Equivalence Across Union)
```sql
SELECT x, y FROM (
SELECT floor(random() * 3) AS x, floor(random() * 3) AS y FROM
generate_series(1, 10)
UNION ALL
SELECT floor(random() * 3) AS x, floor(random() * 3) AS y FROM
generate_series(1, 10)
) ORDER BY x, y;
```
- On `main` (without #27347), planning fails with `SanityCheckPlan`:
```text
SanityCheckPlan: ... does not satisfy order requirements: [x@0 ASC NULLS
LAST, y@1 ASC NULLS LAST]. Child-0 order: [[x@0 ASC NULLS LAST]]
```
- With #27347 (which preserves union equivalences), planning succeeds but
the output is out of order as in Case 1.
### Expected behavior
Volatile expressions should not establish equivalence between projected
columns. `ORDER BY x, y` should sort by both `x` and `y`.
### Additional context
Follow-up to #27347 (see review:
https://github.com/apache/datafusion/pull/27347#pullrequestreview-5902255551).
In `datafusion/physical-expr/src/equivalence/class.rs:770`,
`EquivalenceGroup::project` unconditionally merges all projected targets of the
same source expression into `new_classes` without checking
`!is_volatile(source)`.
### Willingness to contribute
I plan to contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
