sunchao commented on PR #5469: URL: https://github.com/apache/datafusion-comet/pull/5469#issuecomment-5457934428
@comphead Thanks for the pointer. I checked [the SPARK-54918 patch](https://github.com/apache/spark/commit/a4aa4adf4fd0f790d7b71b322292050f6cf8db38). It extends normalization to array set operations such as `array_distinct` and `array_union`. For the scalar sorting and ranking cases here, Spark already treats `-0.0` and `+0.0` as equal before 4.2. The problem is that Comet’s native comparison can distinguish them, splitting rows that should share a rank and potentially dropping qualifying rows at a rank cutoff. This PR normalizes the comparison keys consistently across sorting and ranking, so the fix is still needed with Spark 4.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
