pushnanashi2 opened a new issue, #25071:
URL: https://github.com/apache/datafusion/issues/25071
### Describe the bug
In DataFusion 55.0.0, built from commit
a5c809f98dedcf4b22f5d317ea8efe0720834925, enabling
datafusion.optimizer.enable_unions_to_filter changes the result of a valid
UNION DISTINCT query.
The query returns [1100, 1200] with the option disabled and [1000] with the
option enabled. No error or warning is produced.
The option defaults to false, so the default configuration is not affected.
The impact is incorrect query results when this opt-in optimizer rule is
enabled.
### To Reproduce
The following was reproduced with datafusion-cli built from commit
a5c809f98dedcf4b22f5d317ea8efe0720834925.
CREATE TABLE prices AS
SELECT 1000 AS amount;
SET datafusion.optimizer.enable_unions_to_filter = true;
SELECT prices.amount
FROM (
SELECT amount + 100 AS amount
FROM prices
) AS prices
WHERE amount > 0
UNION
SELECT prices.amount
FROM (
SELECT amount + 200 AS amount
FROM prices
) AS prices
WHERE amount > 1;
The derived tables deliberately reuse the base table name as their alias.
Run the same SELECT with enable_unions_to_filter set to false as a control.
Observed results:
enable_unions_to_filter = false:
1100
1200
enable_unions_to_filter = true:
1000
With the option enabled, EXPLAIN shows that the UNION and both computed
projections have disappeared:
Aggregate: groupBy=[[amount]], aggr=[[]]
Filter: prices.amount > Int64(0) OR prices.amount > Int64(1)
TableScan: prices projection=[amount]
### Expected behavior
The query should return the same two rows regardless of the optimizer
setting:
1100
1200
The optimizer should preserve amount + 100 and amount + 200 before merging
the branch predicates.
### Additional context
The cause appears to be the interaction between extract_branch and
strip_passthrough_nodes in unions_to_filter.rs.
Projections above the branch filter are retained, while projections below
the filter are removed without checking their expressions. In this query, both
branches are reduced to the same TableScan: prices and the same outer
projection, so they are merged into one scan with the combined predicate.
Relevant source:
https://github.com/apache/datafusion/blob/a5c809f98dedcf4b22f5d317ea8efe0720834925/datafusion/optimizer/src/unions_to_filter.rs#L161-L210
https://github.com/apache/datafusion/blob/a5c809f98dedcf4b22f5d317ea8efe0720834925/datafusion/optimizer/src/unions_to_filter.rs#L281-L293
I searched existing DataFusion issues and pull requests for unions_to_filter
and the computed-projection case but did not find a matching report.
I investigated the source code and ran the reproducer myself. I used AI to
cross-check my findings and proofread the English because I am not a native
English speaker.
A regression test should cover two branches over the same base table with
different computed projections below their filters, with the optimization both
enabled and disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
