[
https://issues.apache.org/jira/browse/ARROW-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17612539#comment-17612539
]
Joris Van den Bossche commented on ARROW-17535:
-----------------------------------------------
On the actual issue:
bq. I think in order to enable EXTENSION we need to be able to call to_pylist
or similar on the original extension array from C++ code, in case there were
user provided overrides
For other list arrays, we actually do not convert to lists but to numpy arrays:
{code}
In [3]: pa.array([[1, 2], [3, 4, 5]]).to_numpy(zero_copy_only=False)
Out[3]: array([array([1, 2]), array([3, 4, 5])], dtype=object)
In [4]: pa.array([[1, 2], [3, 4, 5]]).to_pandas().values
Out[4]: array([array([1, 2]), array([3, 4, 5])], dtype=object)
{code}
So it could also be an option to keep using arrays, instead of using lists in
case of ExtensionType. And then, if we can somehow convert the list array's
values to a single array (calling into the Python {{to_numpy}} or
{{to_pandas}}, since that can be overridden?), then we could continue slicing
this into pieces and put that in the resulting array, as we do now (I think
this is basically the first possibility that [~changhiskhan] mentions?)
bq. 2. If the ExtensionType is associated with an ExtensionScalar, then you can
call to_polish on the values array and then use the offsets to construct the
pandas array
That actually brings up a question: if an ExtensionType defines an
ExtensionScalar (but not an associciated pandas dtype, or custom to_numpy
conversion), should we use this scalar's {{as_py()}} for the to_numpy/to_pandas
conversion as well for plain extension arrays? (not the nested case)
Because currently, if you have an ExtensionArray like that (for example using
the example from the docs:
https://arrow.apache.org/docs/dev/python/extending_types.html#custom-scalar-conversion),
we still use the storage type conversion for to_numpy/to_pandas, and only use
the scalar's conversion in {{to_pylist}}.
> [Python] List<Extension> arrays aren't supported in to_pandas calls
> -------------------------------------------------------------------
>
> Key: ARROW-17535
> URL: https://issues.apache.org/jira/browse/ARROW-17535
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, Python
> Reporter: Micah Kornfield
> Priority: Minor
>
> EXTENSION is not in the list of types allowed. I think in order to enable
> EXTENSION we need to be able to call to_pylist or similar on the original
> extension array from C++ code, in case there were user provided overrides.
> Off the top of my head one way of doing this would be to pass through an
> additional std::unorderd_map<Array*, PyObject*> where PyObject is the bound
> to_pylist python function. Are there other alternative that might be cleaner?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
