sam-1112 opened a new pull request, #5802: URL: https://github.com/apache/datafusion-comet/pull/5802
## Which issue does this PR close? Closes #5590. ## Rationale for this change On Spark 4.x, `CometMapSort` currently falls back the entire projection or shuffle to Spark when native `map_sort` does not support the map key type. Spark's `MapSort.doGenCode` supports these cases, and the JVM codegen dispatcher can execute the generated code within the Comet pipeline. This allows the surrounding Comet operator to remain enabled without treating complex-key `MapSort` as a native expression. ## What changes are included in this PR? This PR mixes `CodegenDispatchFallback` into `CometMapSort`. The resulting routing behavior is: - Natively supported scalar keys continue to use native `map_sort`. - Array and struct keys use the JVM codegen dispatcher. - Float and double keys use the dispatcher when `spark.comet.exec.strictFloatingPoint=true`. - If the dispatcher is disabled or cannot handle the expression, execution safely falls back to Spark. The existing `Unsupported`, `Incompatible`, and strict floating-point checks remain unchanged. This PR does not expand `supportedScalarSortElementType` and does not modify the Rust implementation, JNI, protobuf, or wire format. Complex-key `MapSort` is not executed natively; it runs Spark-generated code inside the Comet pipeline. The compatibility documentation and shuffle comments are also updated to distinguish the native scalar-key path from the dispatcher path for other orderable key types. ## How are these changes tested? The Spark 4.x tests were expanded to cover: - Array and struct map keys using the JVM codegen dispatcher. - Strict floating-point keys using the dispatcher. - Scalar map keys continuing to use native `map_sort`. - Safe Spark fallback when the dispatcher is disabled. - Native and columnar shuffles remaining in Comet for array and struct map keys, with `MapSort` routed through the dispatcher. - Query result parity with Spark. Native shuffle tests now expect one Comet shuffle for non-scalar map keys and cover nested hash partitioning with both array and struct keys. Columnar shuffle expectations are updated only for complex map keys; complex map value cases remain unchanged. A collated-string map-key SQL test is not included because Spark rejects such keys during analysis with `COLLATIONS_IN_MAP_KEYS`. The same `Unsupported`-to-dispatcher routing is covered by the array and struct key tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
