skambha commented on issue #24593: [SPARK-27692][SQL] Add new optimizer rule to evaluate the deterministic scala udf only once if all inputs are literals URL: https://github.com/apache/spark/pull/24593#issuecomment-504118807 > Hm, why don't we block (and close) this one for now by SPARK-27761? SPARK-27761 is for the broader discussion of whether we want ScalaUDF to be non deterministic by default or not. 1. From the time we have Scala UDF support in Spark, it has always been deterministic by default and we make decisions in the SQL engine already based on that. 1. The optimization and the idea in this PR is **relevant** irrespective of what we decide to be the default for the Scala UDF. 1. The optimization in this PR is helpful for many scenarios and we use them internally and it boosts performance for relevant queries ( as expected). 1. The PR now also places this optimization under a flag which defaults to false to **minimize any risk** for users who may not want to make use of this optimization by default. Given that a) the changes are fairly minimal, a few lines of code for the optimization itself and b) the risk is minimal as it is under a feature flag, it would be good to get the changes in. WDYT? Please share your comments/concerns on the best way forward. Thanks.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
