Weijun-H opened a new pull request, #6968: URL: https://github.com/apache/datafusion-comet/pull/6968
## Which issue does this PR close? Follow-up to #6476. The existing fallback fixed its incorrect results; this change runs those sort keys natively. RANGE frames that search boundaries over nullable nested keys continue to use Spark as in #6477. ## Rationale for this change Spark applies `NULLS FIRST` or `NULLS LAST` to the root sort key. It compares null elements inside arrays and null fields inside structs according to the sort direction. Native sorting applied the root null choice recursively, so the existing safeguard sent `ASC NULLS LAST` and `DESC NULLS FIRST` on nullable nested keys to Spark. The default `array_sort` comparator has a related distinction: null elements go last while nulls inside nested values compare below non-null values. It currently uses the JVM codegen dispatcher even though Comet already has Spark's recursive comparator. ## What changes are included in this PR? - Wrap affected native sort keys as `struct(isnull(key), key)`, using the sort direction's default null placement for nested comparisons. Keep nondeterministic keys on Spark because wrapping would evaluate the key twice. - Run default `array_sort` through a native function that uses the existing Spark comparator for nested values and puts null elements last. Custom comparators and arrays with nondefault string collations continue through the JVM dispatcher. - Update the SQL fixture, native plan/result tests, and compatibility/tuning documentation. RANGE boundary-search fallback remains in place. ## How are these changes tested? - `make core` - `./mvnw test -Dtest=none -Dsuites="org.apache.comet.CometExpressionSuite nested sort keys keep root"` (Spark 4.1, JDK 17) - `./mvnw test -Pspark-3.5 -Dtest=none -Dsuites="org.apache.comet.CometExpressionSuite nested sort keys keep root"` (Spark 3.5, JDK 17) - `./mvnw test -Dtest=none -Dsuites="org.apache.comet.CometExpressionSuite default array_sort orders null elements last and nested nulls first in native"` (Spark 4.1, JDK 17, with JVM codegen dispatcher disabled in the test) - `./mvnw test -Dtest=none -Dsuites="org.apache.comet.CometArrayExpressionSuite default array_sort with collated strings keeps Spark comparison"` (Spark 4.1, JDK 17, flat and nested collated strings) - `./mvnw test -Dtest=none -Dsuites="org.apache.comet.CometSqlFileTestSuite sort_nested_null_order"` (Spark 4.1, JDK 17) - `cargo test -p datafusion-comet-spark-expr default_array_sort_puts_null_elements_last_and_keeps_null_rows` - `cargo clippy -p datafusion-comet-spark-expr --lib -- -D warnings` - `cargo fmt --all --check`, `./mvnw spotless:check -DskipTests`, `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
