manuzhang opened a new pull request, #4792: URL: https://github.com/apache/datafusion-comet/pull/4792
## Which issue does this PR close? Closes #4719 ## Rationale for this change Spark's exact `Percentile` uses full-precision linear interpolation, while DataFusion's `percentile_cont` quantizes the interpolation weight to 6 decimal places. That can produce visible mismatches for deeply interpolated values, so the supported Comet percentile path had to remain behind `allowIncompatible`. ## What changes are included in this PR? - Add a Comet-native `SparkPercentile` aggregate UDAF that stores values in the existing `List<Float64>` state shape and computes Spark-compatible full-precision interpolation. - Wire Spark `Percentile` planning to the new UDAF instead of DataFusion `percentile_cont`. - Mark the supported single literal percentage/default frequency/numeric/ascending form as compatible by default. - Update benchmark comments and SQL fixtures; add precision regressions for `percentile` and Spark 4 `percentile_cont ... WITHIN GROUP`. ## How are these changes tested? - `cargo fmt --all` - `cargo check -p datafusion-comet-spark-expr` - `cargo check -p datafusion-comet` - `cargo test -p datafusion-comet-spark-expr percentile` - `cargo clippy -p datafusion-comet-spark-expr --all-targets --all-features` - `make core` - `./mvnw test -Pjdk17 -Dtest=none -Dsuites="org.apache.comet.CometSqlFileTestSuite percentile"` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
