Revanth14 opened a new issue, #1509: URL: https://github.com/apache/arrow-go/issues/1509
### Describe the enhancement requested `parquet/variant` has no way to compare two `variant.Value`s. Comparing `Metadata().Bytes()` and `Bytes()` is the only option, and it is stricter than the encoding requires. Two encodings of the same value can differ in their bytes and still be the same value: - the metadata dictionaries differ (another key order, unused keys, unsorted versus sorted); - an object's field values are laid out in a different order (the spec only requires the field IDs to be sorted by key); - the offset or field-ID widths differ; - one string uses the short-string encoding and the other the long-string encoding. iceberg-go compares variant literals by bytes today and leaves a TODO for an upstream method ([literals.go](https://github.com/apache/iceberg-go/blob/main/literals.go#L1644)). **Proposal** Add `func (v Value) Equal(other Value) bool` with the same semantics as Iceberg Java's `VariantPrimitive` and `VariantObject` equality: - **Primitives** are equal when `Type()` and the decoded value are both equal. `Type()` already maps short strings to `String`, so short and long encodings compare equal. `int8(1)` and `int16(1)` are not equal, as in Java and arrow-rs. - **Objects** are equal when they have the same keys, each with an equal value. Metadata dictionaries and value layout are ignored. - **Arrays** are equal when they have the same length and each element is equal. The comparison would respect the same maximum nesting depth as the decoder. arrow-rs's recursive `PartialEq` can overflow the stack on deeply nested input (apache/arrow-rs#10360). **Out of scope:** the spec's equivalence classes (for example `int8` 1 equal to a `decimal16` with scale 2 and unscaled value 100). That is a comparison across physical types, more like what a compute kernel needs, and it could be a separate function later. **Questions** 1. Floats and doubles: compare bits, as Java's `Double.equals` does (`NaN` equals `NaN`, and `0.0` doesn't equal `-0.0`), or use Go's `==`? 2. Decimals: should `1.0` and `1.00` (same type, different scale) be unequal, as with Java's `BigDecimal.equals`? 3. Should values that decode to the same instant at a different timestamp precision (`TimestampMicros` versus `TimestampNanos`) stay unequal? I'd keep them unequal because their types differ. I'd like to work on this. ### Component(s) Parquet ### Component(s) Parquet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
