bvolpato opened a new pull request, #24952: URL: https://github.com/apache/datafusion/pull/24952
## Which issue does this PR close? Related to #24258. This covers runtime operands, floating-point literals, and volatile expressions in compound IN-list rewrites. The NULL-semantics changes in that PR remain separate. ## Rationale for this change The optimizer performs set intersection and difference using expression equality. Different expressions can evaluate to the same value, so these rewrites can remove matching rows even when every column is non-nullable: ```sql CREATE TABLE in_list_values(x INT NOT NULL, a INT NOT NULL, b INT NOT NULL) AS VALUES (1, 1, 1); SELECT x FROM in_list_values WHERE x IN (a, 2, 10, 11) AND x IN (b, 5, 12, 13); ``` The query should return `1`; the rewrite produces an empty intersection and returns no rows. Structural comparison also distinguishes positive and negative floating-point zero. Deduplicating volatile operands can change how many times they are evaluated. ## What changes are included in this PR? - Restrict intersection and difference rewrites to non-null, non-floating literals. - Guard compound rewrites against volatile inputs and list items. - Preserve valid union rewrites for nonvolatile runtime expressions. - Add unit regressions for runtime values, signed zero, mixed integer coercion, and volatility, plus runtime-value cases in `predicates.slt`. ## What is the testing strategy for this PR? - All three new unit tests fail against upstream production code at `35f58f53cde2f634c21f6370e385cf1d7e9bc55c` and pass with the fix. - `cargo test --locked -p datafusion-optimizer --lib`: 777 passed. - `cargo fmt --all -- --check` and `git diff --check` passed. - Draft pending all-target/all-feature Clippy, the extended workspace suite, and execution of `predicates.slt`. ## Are there any user-facing changes? Affected compound IN predicates retain matching rows and preserve volatile expression evaluations. No public API changes. ## Downsides Queries excluded by the new guards may perform more comparisons because the optimizer retains their original predicates. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
