sunchao opened a new pull request, #7722: URL: https://github.com/apache/datafusion-comet/pull/7722
## Which issue does this PR close? Closes #6549. Part of #6385. ## Rationale for this change #6992 routed floating-point map keys through JVM codegen dispatch to match Spark. This PR restores Rust execution for top-level `FLOAT` and `DOUBLE` keys in `map_from_arrays` and `map_from_entries`. It follows up on the closed #6682 with a typed kernel that can land independently of the general map builder changes in #5854. On Spark 4.x with normalization enabled, both builders recognize `[0.0, -0.0]` as a duplicate. For distinct keys `[-0.0, 1.5]`, `map_from_arrays` preserves the negative zero because Spark reuses the input arrays. `map_from_entries` builds a fresh map and stores a positive zero. See [Spark's ArrayBasedMapBuilder](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ArrayBasedMapBuilder.scala#L64-L150). ## What changes are included in this PR? - Both expressions use one Rust kernel for floating-point keys. Comet's shared float helpers make all NaNs compare as one key. Signed zeros compare as one key on Spark 4.0+ unless `spark.sql.legacy.disableMapKeyNormalization=true`. - Under `EXCEPTION`, the kernel reports the first invalid key in insertion order. Under `LAST_WIN`, the first occurrence sets the position and the last supplies the value. If a duplicate forces `map_from_arrays` to rebuild a row, the normalization setting applies to every stored key in that row. The kernel shares input buffers when the output can use the original slices. - Each task receives the resolved settings, so a second action on the same Dataset picks up a change to the normalization flag. For the separate `spark.sql.mapKeyDedupPolicy` setting, Spark whole-stage codegen can retain the first action's policy while Comet reads the current one. The expression audit documents that distinction. - Struct and array keys containing a float use codegen dispatch. I used the repository's `implement-comet-expression` guide for the implementation. ## How are these changes tested? - Parquet-backed SQL fixtures assert Rust execution for both builders and key widths, compare stored keys and ordered values with Spark, and check `DUPLICATED_MAP_KEY` and `NULL_MAP_KEY`. A mixed batch under `LAST_WIN` carries nullable arrays of strings and has one row with a duplicate. - A Scala regression runs the same plan twice across a change to `spark.sql.legacy.disableMapKeyNormalization`. Rust tests inspect key and value bits directly, including noncanonical NaNs and signed zeros, with sliced inputs and null entries. - Local validation covers formatting and `git diff --check`. Hosted CI handles builds and tests, including `run-all-spark-profiles` for Spark 3.5 and 4.x. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
