LuciferYang commented on PR #58845: URL: https://github.com/apache/spark/pull/58845#issuecomment-5957966069
Closing this. Before the change the results were already correct: the definition was evaluated once per reference instead of once per row pair, and analysis keeps a nondeterministic expression out of every join condition, so the extra evaluation never changes an answer. The saving is narrow. Only `BETWEEN` and `NULLIF` build a `With`, and the branch this changes is reached only by a definition that reads both sides of a join, is referenced more than once and is not cheap, such as `ON f(l.x, r.y) BETWEEN a AND b`. The cost is that a `With` left in a join condition is something every rule matching on predicate shapes has to learn: `ReorderJoin`, constraint propagation, `StreamingJoinHelper` and `ExtractEquiJoinKeys` each needed handling here. That is a lot of surface for this saving. The two TODOs this was meant to address are handled separately in SPARK-59955: one describes a case analysis already rules out, and SPARK-58818 already did what the other asks. Looking at that code also turned up SPARK-59954, which I'm fixing first. I'll resolve SPARK-59494 as Won't Fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
