sunchao opened a new pull request, #7722:
URL: https://github.com/apache/datafusion-comet/pull/7722

   ## Which issue does this PR close?
   
   Closes #6549. Part of #6385.
   
   ## Rationale for this change
   
   #6992 routed floating-point map keys through JVM codegen dispatch to match 
Spark. This PR restores Rust execution for top-level `FLOAT` and `DOUBLE` keys 
in `map_from_arrays` and `map_from_entries`. It follows up on the closed #6682 
with a typed kernel that can land independently of the general map builder 
changes in #5854.
   
   On Spark 4.x with normalization enabled, both builders recognize `[0.0, 
-0.0]` as a duplicate. For distinct keys `[-0.0, 1.5]`, `map_from_arrays` 
preserves the negative zero because Spark reuses the input arrays. 
`map_from_entries` builds a fresh map and stores a positive zero. See [Spark's 
ArrayBasedMapBuilder](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ArrayBasedMapBuilder.scala#L64-L150).
   
   ## What changes are included in this PR?
   
   - Both expressions use one Rust kernel for floating-point keys. Comet's 
shared float helpers make all NaNs compare as one key. Signed zeros compare as 
one key on Spark 4.0+ unless `spark.sql.legacy.disableMapKeyNormalization=true`.
   - Under `EXCEPTION`, the kernel reports the first invalid key in insertion 
order. Under `LAST_WIN`, the first occurrence sets the position and the last 
supplies the value. If a duplicate forces `map_from_arrays` to rebuild a row, 
the normalization setting applies to every stored key in that row. The kernel 
shares input buffers when the output can use the original slices.
   - Each task receives the resolved settings, so a second action on the same 
Dataset picks up a change to the normalization flag. For the separate 
`spark.sql.mapKeyDedupPolicy` setting, Spark whole-stage codegen can retain the 
first action's policy while Comet reads the current one. The expression audit 
documents that distinction.
   - Struct and array keys containing a float use codegen dispatch. I used the 
repository's `implement-comet-expression` guide for the implementation.
   
   ## How are these changes tested?
   
   - Parquet-backed SQL fixtures assert Rust execution for both builders and 
key widths, compare stored keys and ordered values with Spark, and check 
`DUPLICATED_MAP_KEY` and `NULL_MAP_KEY`. A mixed batch under `LAST_WIN` carries 
nullable arrays of strings and has one row with a duplicate.
   - A Scala regression runs the same plan twice across a change to 
`spark.sql.legacy.disableMapKeyNormalization`. Rust tests inspect key and value 
bits directly, including noncanonical NaNs and signed zeros, with sliced inputs 
and null entries.
   - Local validation covers formatting and `git diff --check`. Hosted CI 
handles builds and tests, including `run-all-spark-profiles` for Spark 3.5 and 
4.x.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to