asugranyes commented on PR #53695: URL: https://github.com/apache/spark/pull/53695#issuecomment-4520156487
Thanks @cloud-fan this is really helpful. The architectural argument makes sense to me, NormalizeFloatingNumbers being an optimizer rule is the cleaner home for this than carrying normalization logic in the individual operator paths. Good catch on the test too. Comparing with == doesn’t actually verify the normalization claim. I’ll tighten it with doubleToRawLongBits and add the cases you suggested (-0.0 first, symmetric array_except, NaN interaction) when I push the reworked version. Working on it now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
