Hi everyone, I was reviewing a PR (https://github.com/apache/arrow-rs/pull/10417) for an issue in arrow-rs - https://github.com/apache/arrow-rs/issues/10416 - and discovered an inconsistency between the Arrow spec and the Parquet spec.
The issue proposed adding Unsigned integer typed_value support for Shredded Variant based on the Arrow spec - https://arrow.apache.org/docs/format/CanonicalExtensions.html#parquet-variant It says - "A field named typed_value which can be a Primitive Type Mappings or a List, LargeList, ListView or Struct". Where Primitive Type Mapping is https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings This table is inconsistent with the Shredded Value Types table in the Parquet Shredding Spec - https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types Among other differences, the Arrow table allows Uint8/16/32 and Null typed_value columns, which have no valid Parquet shredded representation and can never round trip through Parquet. Implementations have already diverged over this: arrow-rs rejects unsigned typed_value, while Arrow Go accepts it and writes it to Parquet as INT(8/16/32, false), which the shredding spec does not allow. I propose refreshing the Arrow Parquet Variant spec table based on the one in the Parquet Shredding spec. I filed an issue in the Arrow repo with the proposed table - https://github.com/apache/arrow/issues/50622 Thank you, Kosta
