jorisvandenbossche commented on code in PR #33925: URL: https://github.com/apache/arrow/pull/33925#discussion_r1095648801
########## docs/source/format/CanonicalExtensions.rst: ########## @@ -72,4 +72,30 @@ same rules as laid out above, and provide backwards compatibility guarantees. Official List ============= -No canonical extension types have been standardized yet. +Fixed shape tensor +================== + +* Extension name: `arrow.fixed_shape_tensor`. + +* The storage type of the extension: ``FixedSizeList`` where: + + * **value_type** is the data type of individual tensors and + is an instance of ``pyarrow.DataType`` or ``pyarrow.Field``. + * **list_size** is the product of all the elements in tensor shape. + +* Extension type parameters: + + * **value_type** = Arrow DataType of the tensor elements + * **shape** = shape of the contained tensors as a tuple + * **is_row_major** = boolean indicating the order of elements Review Comment: > To me, torch has the ability to change the underlying "physical" ordering depending on how you interpret data. Yes, but both of those use row-major / C-contiguous memory layout. What you change is the order of the dimensions (C-H-W or H-W-C), and to do that while keeping row-major layout, changing between channels first/last layout requires actually shuffling the data in memory (and thus requires a copy). But either "physical order" is row-major and thus can be stored in the proposed FixedShapeTensor array without copy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
