bart-samwel commented on issue #23388: [SPARK-26448][SQL] retain the difference between 0.0 and -0.0 URL: https://github.com/apache/spark/pull/23388#issuecomment-452348776 Let's be consistent with the equals operator now in this PR but then we may want to consider changing that to be consistent with IEEE floating point standard before spark 3.0 as well. Op di 8 jan. 2019 16:51 schreef Wenchen Fan <[email protected]: > Hi @bart-samwel <https://github.com/bart-samwel> , in Spark SQL, > different NaN values are treated as same, -0.0 and 0.0 are treated as same. > So the normalization proposed by this PR keeps the current behavior and > semantic unchanged. > > If later on we find that NaN values should not be treated as same, > basically we need to change 2 places: > > 1. the EQUAL operator should drop the special handling of NaNs, and > always return false for 2 NaNs. > 2. the normalization here should not apply to NaNs. > > Furthermore, even the same NaN value should not equal to itself, so the > binary comparison won't work. Like you proposed we should normalize NaN to > null at that time. > > Anyway, I think we should think about NaN later, as it will be a behavior > change. What do you think? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/spark/pull/23388#issuecomment-452347462>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/ApwVFEqMtsrnpSqA_LjGx-HIDlKZ4rCkks5vBL6GgaJpZM4Zh-ye> > . >
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
