pitrou commented on code in PR #33925:
URL: https://github.com/apache/arrow/pull/33925#discussion_r1094354347
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -72,4 +72,30 @@ same rules as laid out above, and provide backwards
compatibility guarantees.
Official List
=============
-No canonical extension types have been standardized yet.
+Fixed shape tensor
+==================
+
+* Extension name: `arrow.fixed_shape_tensor`.
+
+* The storage type of the extension: ``FixedSizeList`` where:
+
+ * **value_type** is the data type of individual tensors and
+ is an instance of ``pyarrow.DataType`` or ``pyarrow.Field``.
+ * **list_size** is the product of all the elements in tensor shape.
+
+* Extension type parameters:
+
+ * **value_type** = Arrow DataType of the tensor elements
+ * **shape** = shape of the contained tensors as a tuple
+ * **is_row_major** = boolean indicating the order of elements
Review Comment:
Note that "contiguous" can be ambiguous. There are two possible meanings:
* physical contiguity: all values span a single contiguous area in memory
(there are no "holes" without values in them), regardless of how they are
ordered
* logical contiguity: values can be walked in either "C" or "F" order
Numpy uses the latter meaning. Note that logical contiguity implies physical
contiguity, but the converse is not true, for example if you swap some
dimensions in a more than 3+D array. (*)
For now, it seems this spec and discussion is talking about _physical_
contiguity, as it doesn't care how the values are logically accessed (the
interpretation of data is not part of the spec).
(*) Example:
```python
>>> a = np.arange(24).reshape((2, 3, 4))
>>> a
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]]])
>>> a.flags.contiguous
True
>>> a.transpose(0, 2, 1)
array([[[ 0, 4, 8],
[ 1, 5, 9],
[ 2, 6, 10],
[ 3, 7, 11]],
[[12, 16, 20],
[13, 17, 21],
[14, 18, 22],
[15, 19, 23]]])
>>> a.transpose(0, 2, 1).flags.contiguous
False # It's not logically contiguous, but it's physically contiguous!
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
