[
https://issues.apache.org/jira/browse/ARROW-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556210#comment-16556210
]
Antoine Pitrou commented on ARROW-2913:
---------------------------------------
My worry with pinning metadata on the Python side is that the metadata is lost
if you recreate the Python buffers after passing them through some C++
processing chain. This might not be very important, though, and perhaps we can
make this a best-effort thing.
> [Python] Exported buffers don't expose type information
> -------------------------------------------------------
>
> Key: ARROW-2913
> URL: https://issues.apache.org/jira/browse/ARROW-2913
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++, Python
> Affects Versions: 0.10.0
> Reporter: Antoine Pitrou
> Priority: Major
>
> Using the {{buffers()}} method on array gives you a list of buffers backing
> the array, but those buffers lose typing information:
> {code:python}
> >>> a = pa.array(range(10))
> >>> a.type
> DataType(int64)
> >>> buffers = a.buffers()
> >>> [(memoryview(buf).format, memoryview(buf).shape) for buf in buffers]
> [('b', (2,)), ('b', (80,))]
> {code}
> Conversely, Numpy exposes type information in the Python buffer protocol:
> {code:python}
> >>> a = pa.array(range(10))
> >>> memoryview(a.to_numpy()).format
> 'l'
> >>> memoryview(a.to_numpy()).shape
> (10,)
> {code}
> Exposing type information on buffers could be important for third-party
> systems, such as Dask/distributed, for type-based data compression when
> serializing.
> Since our C++ buffers are not typed, it's not obvious how to solve this.
> Should we return tensors instead?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
