jorisvandenbossche commented on issue #38868: URL: https://github.com/apache/arrow/issues/38868#issuecomment-1840652464
The reason I am hesitant to add a direct `FixedShapeTensorArray.__dlpack__` is because this would change the dimensionality of the returned object. All our arrays (including `FixedShapeTensorArray`) are semantically 1D arrays, and all the primitive arrays that will support `__dlpack__` with https://github.com/apache/arrow/pull/38472 also return a 1D tensor (for example, `np.from_dlpack(pa_arr)` will be 1D). Similarly, `Array.to_numpy()` also always gives you a 1D numpy array (including for `FixedShapeTensorArray`). I think it would be inconsistent of such methods like `to_numpy()`, `np.asarray(..)` or `np.from_dlpack(..)` would start to give an object of different dimensionality depending on the data type. --- An alternative way to expose the full nD array backing the `FixedShapeTensorArray` could also be to implement a `to_tensor()` method, which would be a specific API for fixed shape tensor and exactly meant to get the underlying nD array in a zero-copy way. If we then add dlpack support to our Tensor class (which we can do anyway, I think), this would still expose the data of `FixedShapeTensorArray` through DLPack. One might argue that a downside of this is that you then need do an additional `to_tensor()` that you don't need for other types, before being able to convert it to some tensor library with `from_dlpack`. However, I think the user / library will always need to be aware they are handling a fixed shape tensor array: 1) dlpack only works for certain dtypes, so generally you will have to check that anyway, and 2) exactly because it returns a different dimension (and is a very specific data type), I think you are typically handling this specifically already compared to other columns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
