HuaHuaY opened a new issue, #10416:
URL: https://github.com/apache/arrow-rs/issues/10416
### Describe the bug
The Arrow canonical extension specification for `arrow.parquet.variant`
allows
`UInt8`, `UInt16`, and `UInt32` as Arrow `typed_value` storage types, with
the
following logical Variant mappings:
| Arrow `typed_value` type | Variant primitive type |
|--------------------------|------------------------|
| `UInt8` | `Int16` |
| `UInt16` | `Int32` |
| `UInt32` | `Int64` |
However, `parquet-variant-compute` currently rejects these unsigned Arrow
types
when constructing or shredding a `VariantArray`.
There is also an Arrow/Parquet schema boundary that needs special handling:
Parquet itself supports unsigned integer logical types, but the Parquet
Variant
shredding specification does not allow unsigned `typed_value` columns.
Therefore,
canonical unsigned Arrow storage must be lowered to the corresponding signed
Parquet shredded type when writing:
| Arrow `typed_value` | Parquet Variant shredded type |
|---------------------|--------------------------------|
| `UInt8` | `INT32` with `INTEGER(16, true)` |
| `UInt16` | plain `INT32` |
| `UInt32` | plain `INT64` |
Without Variant-specific lowering, a manually constructed Arrow Variant
schema
can be written using Parquet `UINT_*` annotations and subsequently rejected
by
the Parquet Variant reader.
### To Reproduce
```rust
use arrow_schema::DataType;
use parquet_variant::Variant;
use parquet_variant_compute::{shred_variant, VariantArray};
let input = VariantArray::from_iter([
Variant::Int16(0),
Variant::Int16(i16::from(u8::MAX)),
]);
let shredded = shred_variant(&input, &DataType::UInt8)?;
```
### Expected behavior
VariantArray and shred_variant should accept canonical Arrow UInt8, UInt16,
and UInt32 typed_value storage.
### Additional context
Arrow's documentation:
https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
