rednaxelafx opened a new pull request #28008: [WIP][SPARK-XXXXX][SQL] Constant fold deterministic Scala UDFs with foldable arguments URL: https://github.com/apache/spark/pull/28008 ### What changes were proposed in this pull request? Constant fold deterministic Scala UDFs with foldable arguments, conservatively. `ScalaUDF`s that meet all following criteria are subject to constant folding in this PR: - deterministic - all arguments are foldable - does not throw an exception when evaluating the UDF for constant folding ### Why are the changes needed? This is an optimization that enables more constant folding, improving the performance of a special case of Scala UDFs. Catalyst already implements constant folding of expressions in a few places, through the pattern `if (e.foldable) e.eval(EmptyRow)`. One of the optimizer rules that specifically performs constant folding is `ConstantFolding`. `ScalaUDF` does not override `foldable`, so it isn't subject to constant folding right now. To enable constant folding of `ScalaUDF`s, it's tempting to make it override `foldable`: ```scala override lazy val foldable: Boolean = deterministic && children.forall(_.foldable) ``` But `ScalaUDF`s are declared as deterministic by default, so it's possible for users to mis-declare a potentially exception-throwing UDF as deterministic, so for callers of the "constant folding" code pattern that do not expect an exception, overriding `ScalaUDF.foldable` in general may have a wide impact. Instead, this PR tackles the problem in a conservative way, where the constant folding of `ScalaUDF`s is only implemented in the `ConstantFolding` rule, with special handling to catch non-fatal exceptions when evaluating a Scala UDF during constant folding and skip folding such UDFs. ### Does this PR introduce any user-facing change? No. Caveat: this PR assumes certain semantics of `Expression.deterministic`. Although it guards against potentially exception-throwing Scala UDFs being mis-declared as deterministic, it cannot guard against other kinds of mis-declarations, e.g. - UDFs that are only expected to be invoked on executors; - UDFs that return the same value but have other side-effects, such as modifying a static field. Users should explicitly mark such UDFs as `asNondeterministic`. ### How was this patch tested? Added a test case in `ConstantFoldingSuite`.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
