sarutak commented on PR #59940: URL: https://github.com/apache/spark/pull/59940#issuecomment-6634680110
Thank you for the third round, @dongjoon-hyun. Items 1 and 2 reproduce as you described: the division fails with `DIVIDE_BY_ZERO`, and the Python UDF query returns the 29 rows. I pushed 3675391a312; replies are inline. In short: the conditions are now classified in one loop (item 5), which keeps a non-deterministic condition above after a condition over a Python UDF (item 2). For item 1, I took the kept `IS NOT NULL` check out again: with whole-stage codegen, keeping it above can not keep the conditions written after it behind it, so the inferred checks are better handled where they are inferred, as you suggest in item 3, which I would like to do in a separate PR. The migration guide entry is down to four sentences (item 6). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
