sdf-jkl commented on PR #10639: URL: https://github.com/apache/arrow-rs/pull/10639#issuecomment-5458889280
This is a more difficult one because it is still an open question and has a long history. My head is spinning after thinking about it all day. The Arrow canonical extension documentation does not list `FixedSizeList` as a supported `typed_value` type. However, I could not find `FixedSizeList` being considered or mentioned during the discussion in apache/arrow#47456, so I am hesitant to treat its omission as a deliberate decision to reject it. At the Parquet level, an Arrow `FixedSizeList` is still encoded as an ordinary three-level `LIST`. The fixed length exists only in the Arrow schema metadata, so accepting it is not obviously a question of Parquet Variant specification compliance. The relevant arrow-rs history is: - #8241 introduced the `ListLikeArray` trait for `cast_to_variant`. - #8282 implemented it for `FixedSizeListArray`. - #8514 reused it to add unshredding support for all five Arrow list-like types, including `FixedSizeList`. - #9663 added the `FixedSizeList` builder used by `variant_to_arrow`, `variant_get`, and `shred_variant`. Using `ListLikeArray` for `cast_to_variant` makes sense: Variant value encoding has only ordinary arrays, so an Arrow `FixedSizeList` becomes a Variant array and its fixed length is discarded. I think reusing the same abstraction to determine which types `unshred_variant` accepts was a mistake. `ListLikeArray` only says that an Arrow array can expose child values and an element range; it does not say that every implementing type is a valid shredded `typed_value`. If we decide that `FixedSizeList` is not a supported Variant `typed_value`, then both `shred_variant` and `unshred_variant` should reject it. Keeping the FSL unshredding branch does not currently provide backwards-compatible reading because `VariantArray::try_new` already rejects an FSL `typed_value` before `unshred_variant` can be called. Alternatively, if we want a lenient-reader policy for data previously produced by arrow-rs, we should explicitly accept or normalize FSL on read and add a persisted read/unshred test. Leaving only the existing internal unshredding branch does not accomplish that. DuckDB provides the clearest comparison. Its fixed-length list equivalent, `ARRAY`, is explicitly rejected as a shredding target. Its Variant reader has no FSL-specific unshredding path: the physical Parquet `LIST` is inferred as a variable-length DuckDB `LIST`, and Arrow-specific fixed-size metadata is not restored. It therefore unshreds the value as an ordinary Variant array without preserving or validating a fixed length. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
