raghuvanshraj opened a new issue, #12302: URL: https://github.com/apache/arrow-rs/issues/12302
**Is your feature request related to a problem or challenge?** In datafusion-variant, `GROUP BY` and `JOIN` on a Variant value return wrong results (datafusion-contrib/datafusion-variant#183). The engine hashes and compares the `metadata` + `value` bytes, and logically equal values do not share bytes. Pulling `"api"` out of three rows with different field sets gives three different `metadata` buffers, so three groups instead of one. I looked at what `VariantBuilder` gives me if I just rebuild a value through it, hoping that would be enough. It is not. The dictionary keeps insertion order and only tracks whether that order happened to be sorted ([metadata.rs#L164-L176](https://github.com/apache/arrow-rs/blob/e37fdec8691c5a1c51ea14b9ee89b496cd749963/parquet-variant/src/builder/metadata.rs#L164-L176)), and an `Int64(1)` is written back as an 8-byte int rather than the smallest width that fits. Negative zero, NaN bit patterns and decimal scale are kept as-is too. A `variant_normalize` built on rebuilding was removed from datafusion-variant for exactly this reason (datafusion-contrib/datafusion-variant#119). **Describe the solution you'd like** A `canonicalize` operation in `parquet-variant` that returns the same Variant value with a single, stable byte encoding, following the rules Spark is adopting in apache/spark#59669: - metadata: only the field names actually used, sorted by unsigned UTF-8 bytes, minimal offset width - objects: fields written in key order; arrays keep element order - integers, field ids and offsets: smallest encoding that fits - decimals: trailing zeros stripped; integral decimals that fit become integers - floats: one representation for -0.0 and for NaN, keeping FLOAT vs DOUBLE An already canonical input comes back unchanged. With this, two values that are equal under the Variant equality rules have equal bytes, so a byte hash and a byte compare are enough for grouping, joins and `DISTINCT`. I'd like to work on this. **Describe alternatives you've considered** A logical hash alongside the existing logical `PartialEq`. That would fix equality checks but not query engines that hash the raw bytes of the struct, which is what DataFusion does for group and join keys. **Additional context** datafusion-contrib/datafusion-variant#183 is the user-facing symptom; the plan there is to wrap Variant keys in this canonicalize before hashing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
