peterxcli opened a new pull request, #5041: URL: https://github.com/apache/datafusion-comet/pull/5041
## Which issue does this PR close? Closes #4131. ## Rationale for this change DataFusion LAST_VALUE partial-state merging filtered unset states but selected element 0 from the final state array instead of its last element. This could make Comet produce a different result from Spark, so Comet temporarily fell back for FIRST/LAST in PartialMerge mode. The root fix is [apache/datafusion#23905](https://github.com/apache/datafusion/pull/23905). Because Comet currently depends on DataFusion 54.1.0 and the upstream PR is based on a newer, incompatible DataFusion API, this draft uses a 54.1.0 backport at [peterxcli/datafusion@f594b4e79cb969fc1025cb4a91c646cfc4bbae3a](https://github.com/peterxcli/datafusion/commit/f594b4e79cb969fc1025cb4a91c646cfc4bbae3a). ## What changes are included in this PR? - Pin the DataFusion workspace dependencies to the public backport commit. - Remove the temporary FIRST/LAST PartialMerge fallback. - Change the existing FIRST/LAST PartialMerge SQL cases from fallback assertions to Spark/Comet result comparisons. ## How are these changes tested? - `cargo build --locked` - `JAVA_HOME=/Users/lixucheng/.sdkman/candidates/java/17.0.14-zulu make core` - `JAVA_HOME=/Users/lixucheng/.sdkman/candidates/java/17.0.14-zulu DYLD_LIBRARY_PATH=/Users/lixucheng/.sdkman/candidates/java/17.0.14-zulu/lib/server ./mvnw test -Dtest=none -Dsuites="org.apache.comet.CometSqlFileTestSuite partial_merge" -Dscalastyle.skip=true` The focused SQL suite passed both Parquet dictionary configurations: 2 tests run, 2 succeeded. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
