dongjoon-hyun commented on issue #25331: [SPARK-27768][SQL] Infinity, -Infinity, NaN should be recognized in a case insensitive manner. URL: https://github.com/apache/spark/pull/25331#issuecomment-520314155 Sorry for being late, guys. Hi, @srowen . I agree with you that `Any improvement or bug fix is a behavior change`. However, this PR causes a different SQL result which can be considered as a correctness issue to the existing Spark SQL apps. For example, we can imagine a SQL reporting app which processes two rows `nan` and `NaN`. Previously `nan` is considered as `NULL`, but now `nan` become a `NaN`. So, when you do `GROUP BY` on this values, Spark 3.0 will return one row while Spark 2.4 returns two rows. For the infinity values, the situation is the same. In this case, we had better have a SQL migration guide. Otherwise, during Spark migration acceptance tests, SQL result comparison will fail because this kind of corner cases are tested always. Also, cc @gatorsmile since he filed this issue.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
