[
https://issues.apache.org/jira/browse/ARROW-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436220#comment-16436220
]
ASF GitHub Bot commented on ARROW-2451:
---------------------------------------
robertnishihara commented on a change in pull request #1887: ARROW-2451:
[Python] Handle non-object arrays more efficiently in custom serializer.
URL: https://github.com/apache/arrow/pull/1887#discussion_r181203944
##########
File path: python/pyarrow/serialization.py
##########
@@ -37,11 +37,22 @@
# python_to_arrow.cc)
def _serialize_numpy_array_list(obj):
- return obj.tolist(), obj.dtype.str
+ if obj.dtype.str != '|O':
Review comment:
I'm not sufficiently familiar with numpy dtypes to know if this `|O` is the
only relevant special case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Handle more dtypes efficiently in custom numpy array serializer.
> ----------------------------------------------------------------
>
> Key: ARROW-2451
> URL: https://issues.apache.org/jira/browse/ARROW-2451
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Reporter: Robert Nishihara
> Assignee: Robert Nishihara
> Priority: Major
> Labels: pull-request-available
>
> Right now certain dtypes like bool or fixed length strings are serialized as
> lists, which is inefficient. We can handle these more efficiently by casting
> them to uint8 and saving the original dtype as additional data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
