dongjoon-hyun commented on PR #60163: URL: https://github.com/apache/spark/pull/60163#issuecomment-6689985888
Thank you for the review, @spark-test. I ran the queries in the review on this PR with and without #60162. - **Constraint inference**: Yes. As the description says, this PR should be merged after #60162. With it, the queries in the description, including `id > 5 AND y = 0 AND 10 / (id - y) > 0`, return their results. - **CASE/IF**: Confirmed. With #60162, `y = 5 AND CASE WHEN y > 0 THEN x ELSE z END > 1` no longer pushes `x > 1` down and reads `z`. I'd keep the current boundary in this PR. Rewriting through a `CASE WHEN` or `IF` whose conditional parts cannot fail can be a separate improvement. - **#60162 end state**: #60162 does not infer a filter that can fail, so the copies are gone. The UDF in `y = 5 AND f(x, y) > 0` runs once per row (10 calls for 10 rows), and `x > 1` is not inferred from the CASE above. - **Correlated arithmetic**: Confirmed. `l.b = 5.0 AND l.a NOT IN (SELECT c FROM r WHERE r.d + l.b > 10)` becomes a broadcast nested loop join instead of a null-aware broadcast hash join. I'd accept this cost for now, since it needs arithmetic over an outer column that is bound to a literal. I updated the description, which said that these conditions were rewritten as before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
