jorisvandenbossche commented on code in PR #33925: URL: https://github.com/apache/arrow/pull/33925#discussion_r1095684790
########## docs/source/format/CanonicalExtensions.rst: ########## @@ -72,4 +72,30 @@ same rules as laid out above, and provide backwards compatibility guarantees. Official List ============= -No canonical extension types have been standardized yet. +Fixed shape tensor +================== + +* Extension name: `arrow.fixed_shape_tensor`. + +* The storage type of the extension: ``FixedSizeList`` where: + + * **value_type** is the data type of individual tensors and + is an instance of ``pyarrow.DataType`` or ``pyarrow.Field``. + * **list_size** is the product of all the elements in tensor shape. + +* Extension type parameters: + + * **value_type** = Arrow DataType of the tensor elements + * **shape** = shape of the contained tensors as a tuple + * **is_row_major** = boolean indicating the order of elements Review Comment: > I don't know if that helps the discussion: For your example, `x` and `y` indeed have a different layout of the values in memory, while seemingly representing the same tensor (in you print them). But that's because `y` is transposed after it was created (and this only changed the logical order, not the physical), and thus has custom strides. But you could still store both x and y without copy in a FixedShapeTensor array. The difference is that `x` would be stored as shape (2, 3), and y would be stored as (3, 2). And assume that in this dummy example the dimensions are called A and B, and if you want to keep track of the correct logical order, you would store the dimension names for `x` as `["A", "B"]` and for `y` as `["B", "A"]`. Then afterwards, after reading such data and the application knows it always wants the data in (A, B) order, it can transpose the data read from the stored `y` (just as you did when creating `y`) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
