ctagard commented on PR #59403: URL: https://github.com/apache/spark/pull/59403#issuecomment-6170411311
Hey @HyukjinKwon, thank you so much for your detailed feedback! I implemented a fix for the blocker, the IN / NOT IN subqueries over a normalized key skip join-key normalization. I reproduced the issue, and followed your suggestion by updating NormalizeFloatingNumbers which now checks the keys on both sides, in both the equip-join case and the null-aware anti join case. Already normalized keys are left as is, so only the subquery side gets wrapped. I fixed it there rather than adding InSubquery to the exemption in normalizeGroupingExpressions because the exemption would not cover a computed grouping key used as the IN value, and the rule-level fix also covers any other one-sided tag. Added tests to both the NormalizeFloatingPointNumbersSuite and the DataFrameAggregateSuite. Looking forward to any more feedback! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
