raghuvanshraj opened a new issue, #12302:
URL: https://github.com/apache/arrow-rs/issues/12302

   **Is your feature request related to a problem or challenge?**
   
   In datafusion-variant, `GROUP BY` and `JOIN` on a Variant value return wrong 
results (datafusion-contrib/datafusion-variant#183). The engine hashes and 
compares the `metadata` + `value` bytes, and logically equal values do not 
share bytes. Pulling `"api"` out of three rows with different field sets gives 
three different `metadata` buffers, so three groups instead of one.
   
   I looked at what `VariantBuilder` gives me if I just rebuild a value through 
it, hoping that would be enough. It is not. The dictionary keeps insertion 
order and only tracks whether that order happened to be sorted 
([metadata.rs#L164-L176](https://github.com/apache/arrow-rs/blob/e37fdec8691c5a1c51ea14b9ee89b496cd749963/parquet-variant/src/builder/metadata.rs#L164-L176)),
 and an `Int64(1)` is written back as an 8-byte int rather than the smallest 
width that fits. Negative zero, NaN bit patterns and decimal scale are kept 
as-is too. A `variant_normalize` built on rebuilding was removed from 
datafusion-variant for exactly this reason 
(datafusion-contrib/datafusion-variant#119).
   
   **Describe the solution you'd like**
   
   A `canonicalize` operation in `parquet-variant` that returns the same 
Variant value with a single, stable byte encoding, following the rules Spark is 
adopting in apache/spark#59669:
   
   - metadata: only the field names actually used, sorted by unsigned UTF-8 
bytes, minimal offset width
   - objects: fields written in key order; arrays keep element order
   - integers, field ids and offsets: smallest encoding that fits
   - decimals: trailing zeros stripped; integral decimals that fit become 
integers
   - floats: one representation for -0.0 and for NaN, keeping FLOAT vs DOUBLE
   
   An already canonical input comes back unchanged. With this, two values that 
are equal under the Variant equality rules have equal bytes, so a byte hash and 
a byte compare are enough for grouping, joins and `DISTINCT`. I'd like to work 
on this.
   
   **Describe alternatives you've considered**
   
   A logical hash alongside the existing logical `PartialEq`. That would fix 
equality checks but not query engines that hash the raw bytes of the struct, 
which is what DataFusion does for group and join keys.
   
   **Additional context**
   
   datafusion-contrib/datafusion-variant#183 is the user-facing symptom; the 
plan there is to wrap Variant keys in this canonicalize before hashing.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to