rok commented on code in PR #37166:
URL: https://github.com/apache/arrow/pull/37166#discussion_r1348787269
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -148,6 +148,113 @@ Fixed shape tensor
by this specification. Instead, this extension type lets one use fixed shape
tensors
as elements in a field of a RecordBatch or a Table.
+.. _variable_shape_tensor_extension:
+
+Variable shape tensor
+=====================
+
+* Extension name: `arrow.variable_shape_tensor`.
+
+* The storage type of the extension is: ``StructArray`` where struct
+ is composed of **data** and **shape** fields describing a single
+ tensor per row:
+
+ * **data** is a ``List`` holding tensor elements (each list element is
+ a single tensor). The List's value type is the value type of the tensor,
+ such as an integer or floating-point type.
+ * **shape** is a ``FixedSizeList<int32>[ndim]`` of the tensor shape where
+ the size of the list ``ndim`` is equal to the number of dimensions of the
+ tensor.
+
+* Extension type parameters:
+
+ * **value_type** = the Arrow data type of individual tensor elements.
+
+ Optional parameters describing the logical layout:
+
+ * **dim_names** = explicit names to tensor dimensions
+ as an array. The length of it should be equal to the shape
+ length and equal to the number of dimensions.
+
+ ``dim_names`` can be used if the dimensions have well-known
+ names and they map to the physical layout (row-major).
+
+ * **permutation** = indices of the desired ordering of the
+ original dimensions, defined as an array.
+
+ The indices contain a permutation of the values [0, 1, .., N-1] where
+ N is the number of dimensions. The permutation indicates which
+ dimension of the logical layout corresponds to which dimension of the
+ physical tensor (the i-th dimension of the logical view corresponds
+ to the dimension with number ``permutations[i]`` of the physical tensor).
+
+ Permutation can be useful in case the logical order of
+ the tensor is a permutation of the physical order (row-major).
+
+ When logical and physical layout are equal, the permutation will always
+ be ([0, 1, .., N-1]) and can therefore be left out.
+
+ * **uniform_shape** = sizes of individual tensor's dimensions which are
+ guaranteed to stay constant in uniform dimensions and can vary in
+ non-uniform dimensions. This holds over all tensors in the array.
+ Sizes in uniform dimensions are represented with int32 values, while
+ sizes of the non-uniform dimensions are not known in advance and are
+ represented with null. If ``uniform_shape`` is not provided it is assumed
+ that all dimensions are non-uniform.
+ An array containing a tensor with shape (2, 3, 4) and whose first and
+ last dimensions are uniform would have ``uniform_shape`` (2, null, 4).
+ This allows for interpreting the tensor correctly without accounting for
+ uniform dimensions while still permitting optional optimizations that
+ take advantage of the uniformity.
+
+* Description of the serialization:
+
+ The metadata must be a valid JSON object that optionally includes
+ dimension names with keys **"dim_names"** and ordering of dimensions
+ with key **"permutation"**.
+ Shapes of tensors can be defined in a subset of dimensions by providing
+ key **"uniform_shape"**.
+ Minimal metadata is an empty JSON object.
+
+ - Example of minimal metadata is:
+
+ ``{}``
Review Comment:
```suggestion
Minimal metadata is an empty string.
- In case of minimal metadata we assume that all dimensions are non-uniform
and the ordering of dimensions is the same as the physical layout.
Shape of individual tensors is given by ``shape``
``FixedSizeList<int32>[ndim]``
field which also implies number of dimensions with its length ``ndim``.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
