Jim Pivarski commented on ARROW-2295:

I second this and would like to request that the Numpy interface has more 
low-level access to Arrow structures. For instance, ListArray is internally 
represented as two arrays: offsets and contents, and there are applications 
where we'd want to get a zero-copy view of these arrays. The to_pandas() 
function constructs a Numpy object array of subarrays, which is a performance 
bottleneck if you really do want the original offsets and contents.

This function could be an inverse of pyarrow.ListArray.from_arrays, something 
that returns the offsets and contents as Numpy arrays for a List<Primitive> and 
something more complex for general cases (a dict from strings representing a 
place in the hierarchy to Numpy arrays?).

A simpler interface that could be implemented immediately would be one that 
returns the raw bytes of the Arrow buffer, to let us identify its contents 
using [the Arrow 
spec|[https://github.com/apache/arrow/blob/master/format/Layout.md].] But that 
doesn't make use of the dtype (probably just set it to uint8) and would 
probably make more sense as a raw __buffer__. (Should that be a separate JIRA 


> Add to_numpy functions
> ----------------------
>                 Key: ARROW-2295
>                 URL: https://issues.apache.org/jira/browse/ARROW-2295
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Python
>            Reporter: Lawrence Chan
>            Priority: Minor
> There are `to_pandas()` functions, but no `to_numpy()` functions. I'd like to 
> propose that we include both.
> Also, `pyarrow.lib.Array.to_pandas()` returns a `numpy.ndarray`, which imho 
> is very confusing :). I think it would be more intuitive for the 
> `to_pandas()` functions to return `pandas.Series` and `pandas.DataFrame` 
> objects, and the `to_numpy()` functions to return `numpy.ndarray` and either 
> a ordered dict of `numpy.ndarray` or a structured `numpy.ndarray` depending 
> on a flag, for example. The `to_pandas()` function is of course welcome to 
> use the `to_numpy()` func to avoid the additional index and whatnot of the 
> `pandas.Series`.

This message was sent by Atlassian JIRA

Reply via email to