zhangfengcdt opened a new pull request, #6928: URL: https://github.com/apache/datafusion-comet/pull/6928
## Which issue does this PR close? Closes #6799. ## Rationale for this change Spark keeps `0.0` and `-0.0` as separate keys in the `percentile` buffer (SPARK-45599), then stable-sorts them under SQL ordering, where they are equal. So the zero its `OpenHashMap` iterates first takes the lower ranks, and which one that is depends on the table's capacity and collisions. Comet's native percentile picked either zero, so the result could have the wrong sign. ## What changes are included in this PR? - When a group holds both zeros, the native percentile replays Spark's partial, deserialized and final hash tables (murmur3 slots, growth, quadratic probing) to find which zero Spark ranks first. Other inputs only pay one scan for a `-0.0`. - A new `float_input` field on the `Percentile` proto, because Spark hashes `FLOAT` input as a float, before Comet's cast to double. ## How are these changes tested? - Rust unit tests: the SPARK-45599 rows return `-0.0`, a table grown to 1024 slots returns `0.0`, and a lone zero keeps its sign. - New cases in `percentile.sql`: the SPARK-45599 rows, both zero insertion orders per group, and a 502-row table large enough to make Spark's table grow. The large-table case fails without the fix (Spark `0.0`, Comet `-0.0`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
