sdf-jkl opened a new issue, #1202:
URL: https://github.com/apache/arrow-go/issues/1202
### Describe the bug, including details regarding any error messages,
version, and platform.
## Description
Arrow Go permits unsigned Arrow integer types as `typed_value` fields in the
`arrow.parquet.variant` extension type. When such an array is written to
Parquet, the unsigned Arrow type is materialized directly as an unsigned
Parquet integer column.
For example, an Arrow `Uint32` Variant `typed_value` produces:
```text
optional int32 typed_value (Int(bitWidth=32, isSigned=false));
```
This is not a valid shredded Variant type according to the Parquet Variant
shredding specification. Its supported integer representations are:
- `int8` → `INT32 / INT(8, true)`
- `int16` → `INT32 / INT(16, true)`
- `int32` → `INT32`
- `int64` → `INT64`
Unsigned Parquet integer logical types are not supported.
The current Arrow canonical Variant extension specification permits unsigned
Arrow storage types through widening mappings:
- `Uint8` → Variant `Int16`
- `Uint16` → Variant `Int32`
- `Uint32` → Variant `Int64`
## Interoperability test
The parquet-testing interoperability suite includes case 127, which expects
`INTEGER(32,false)` to be rejected:
```json
{
"case_number": 127,
"test": "testUnsignedInteger",
"parquet_file": "case-127.parquet",
"error_message": "Unsupported shredded value type: INTEGER(32,false)"
}
```
Arrow Go currently skips that case with:
```go
s.T().Skip("Skipping case 127: test says uint32 should error, we just upcast
to int64")
```
The Parquet column is not upcast to `int64`. It is materialized as Arrow
`Uint32`, retained as `array.Uint32`, and written back to Parquet as `INT32 /
INT(32,false)`.
Variant reconstruction may encode the value as a signed Variant integer,
using `Int64` where necessary, but that does not make the shredded Parquet
schema valid.
## Expected behavior
At minimum, Arrow Go should cast unsigned Arrow `typed_value` columns to
their signed, widened Variant representations when writing Parquet:
- `Uint8` → signed `Int16`
- `Uint16` → signed `Int32`
- `Uint32` → signed `Int64`
The resulting Parquet `typed_value` columns must use the corresponding
signed shredded types rather than unsigned integer logical annotations.
The more complete resolution is to drop unsigned `typed_value` support from
the Arrow Variant extension type altogether. If the Arrow specification change
below is accepted, Arrow Go can reject unsigned Variant `typed_value` schemas
directly, without requiring fine-grained conversions in the Parquet writer.
The interoperability test for case 127 should no longer be skipped. Its skip
comment should also be removed or corrected because no Parquet schema upcast
currently occurs.
## Related Arrow specification change
The current Arrow canonical extension specification permits these unsigned
mappings, creating a representation that cannot map directly to a conforming
Parquet shredded schema.
The following PR proposes aligning Arrow’s Variant primitive mappings with
the Parquet shredding specification by removing unsupported mappings such as
unsigned integers and Null:
- https://github.com/apache/arrow/pull/50810
- Related issue: https://github.com/apache/arrow/issues/50622
## Relevant links
- [Parquet Variant shredded-value
types](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types)
- [Current Arrow canonical Variant primitive
mappings](https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings)
- [Proposed Arrow specification
correction](https://github.com/apache/arrow/pull/50810)
- [parquet-testing case
127](https://github.com/apache/parquet-testing/blob/e7d32dac5b4cbf017fefd598a03686f53370b292/shredded_variant/cases.json#L754-L757)
- [Arrow Go skips case
127](https://github.com/apache/arrow-go/blob/986b027408e1d973cb78023fc74a2a1bc361c6a9/parquet/pqarrow/variant_test.go#L295-L304)
- [`INTEGER(32,false)` maps to Arrow
`Uint32`](https://github.com/apache/arrow-go/blob/986b027408e1d973cb78023fc74a2a1bc361c6a9/parquet/pqarrow/schema.go#L433-L454)
- [The shredded reader retains
`array.Uint32`](https://github.com/apache/arrow-go/blob/986b027408e1d973cb78023fc74a2a1bc361c6a9/arrow/extensions/variant.go#L646-L663)
- [Arrow Go’s Variant mappings include unsigned
types](https://github.com/apache/arrow-go/blob/986b027408e1d973cb78023fc74a2a1bc361c6a9/arrow/extensions/variant.go#L1195-L1206)
- [The shredded builder accepts unsigned Arrow
builders](https://github.com/apache/arrow-go/blob/986b027408e1d973cb78023fc74a2a1bc361c6a9/arrow/extensions/variant.go#L1456-L1486)
- [The writer emits unsigned Parquet
annotations](https://github.com/apache/arrow-go/blob/986b027408e1d973cb78023fc74a2a1bc361c6a9/parquet/pqarrow/schema.go#L624-L645)
Generated with codex and proof read
### Component(s)
Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
