viirya commented on PR #59944: URL: https://github.com/apache/spark/pull/59944#issuecomment-6517646037
Thanks for the review. Both points are addressed in 6228e8b3a66: - The end-to-end queries in `DataFrameSuite` and `PythonUDFSuite` are now built inside the config loops, so each iteration is planned under its own config. - The `Aggregate` case now checks the alias-substituted condition for `throwable`, so a condition over a throwable result column, such as your `x IS NULL`, stays above when a condition written before it does. A plan test covers it, and so does an end-to-end test under whole-stage codegen. As you noted, with codegen off the aggregate computes `x` for every group anyway. The generic case still checks the condition as written, since it does not see the aliases below the operator. The description now says that a condition over a throwable alias is only covered in the `Project` and `Aggregate` cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
