sdf-jkl commented on code in PR #50810: URL: https://github.com/apache/arrow/pull/50810#discussion_r3926985275
########## docs/source/format/CanonicalExtensions.rst: ########## @@ -488,63 +488,107 @@ binary values look like. Primitive Type Mappings ----------------------- -+----------------------+------------------------+ -| Arrow Primitive Type | Variant Primitive Type | -+======================+========================+ -| Null | Null | -+----------------------+------------------------+ -| Boolean | Boolean (true/false) | -+----------------------+------------------------+ -| Int8 | Int8 | -+----------------------+------------------------+ -| Uint8 | Int16 | -+----------------------+------------------------+ -| Int16 | Int16 | -+----------------------+------------------------+ -| Uint16 | Int32 | -+----------------------+------------------------+ -| Int32 | Int32 | -+----------------------+------------------------+ -| Uint32 | Int64 | -+----------------------+------------------------+ -| Int64 | Int64 | -+----------------------+------------------------+ -| Float | Float | -+----------------------+------------------------+ -| Double | Double | -+----------------------+------------------------+ -| Decimal32 | decimal4 | -+----------------------+------------------------+ -| Decimal64 | decimal8 | -+----------------------+------------------------+ -| Decimal128 | decimal16 | -+----------------------+------------------------+ -| Date32 | Date | -+----------------------+------------------------+ -| Time64 | TimeNTZ | -+----------------------+------------------------+ -| Timestamp(us, UTC) | Timestamp (micro) | -+----------------------+------------------------+ -| Timestamp(us) | TimestampNTZ (micro) | -+----------------------+------------------------+ -| Timestamp(ns, UTC) | Timestamp (nano) | -+----------------------+------------------------+ -| Timestamp(ns) | TimestampNTZ (nano) | -+----------------------+------------------------+ -| Binary | Binary | -+----------------------+------------------------+ -| LargeBinary | Binary | -+----------------------+------------------------+ -| BinaryView | Binary | -+----------------------+------------------------+ -| String | String | -+----------------------+------------------------+ -| LargeString | String | -+----------------------+------------------------+ -| StringView | String | -+----------------------+------------------------+ -| UUID extension type | UUID | -+----------------------+------------------------+ +The following table defines the set of Arrow types that are valid as primitive +``typed_value`` storage. It follows the `Shredded Value Types +<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__ +table of the Parquet Variant Shredding specification. Each row maps a Variant +primitive type to the Parquet `physical type +<https://parquet.apache.org/docs/file-format/types/>`__ and `logical type +<https://parquet.apache.org/docs/file-format/types/logicaltypes/>`__ +annotation of a shredded ``typed_value`` column, and to the Arrow +:ref:`data types <data_types>` covering that Variant type's full value +domain. An empty *Parquet Logical Type* cell means the physical type carries +no explicit annotation. This is intentional for ``int32`` and ``int64``: +Parquet defines ``INT(32, true)`` and ``INT(64, true)`` as implied by the +unannotated ``INT32`` and ``INT64`` physical types, respectively. Review Comment: I added it due to @alamb comment above > I am a little confused about what (logical or physical) type is used in the Parquet Type column. Some seem to be just the physical type (INT64) , and some seem to have both logical and physical (INT32, INT(8, true) ). It would help to be spcific about what the Parquet type column represents and use it consistently The Variant spec omits logical types IN32, INT64 the same way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
