chris-twiner commented on PR #34558: URL: https://github.com/apache/spark/pull/34558#issuecomment-2404981056
> @Kimahriman just out of curiosity, how much did the performance improve? I just wanted to add to the above response that I've implemented a compilation scheme [here](https://sparkutils.github.io/quality/latest/advanced/userFunctions/#controlling-compilation-tweaking-the-quality-optimisations), as part of Quality, and we saw perf boosts of up to 40%, after that adding further lambdas triggered the cost of code generation being higher than the saving. It's definitely usage dependant though, the more work done in the function the higher the cost, a small boost is noticeable on removal of the atomic under similar ideal circumstances. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
