[
https://issues.apache.org/jira/browse/ARROW-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436225#comment-16436225
]
ASF GitHub Bot commented on ARROW-2451:
---------------------------------------
robertnishihara commented on a change in pull request #1887: ARROW-2451:
[Python] Handle non-object arrays more efficiently in custom serializer.
URL: https://github.com/apache/arrow/pull/1887#discussion_r181204836
##########
File path: python/pyarrow/tests/test_serialization.py
##########
@@ -315,9 +315,12 @@ def test_default_dict_serialization(large_buffer):
def test_numpy_serialization(large_buffer):
for t in ["bool", "int8", "uint8", "int16", "uint16", "int32",
- "uint32", "float16", "float32", "float64"]:
+ "uint32", "float16", "float32", "float64", "<U1", "<U2", "<U3",
+ "<U4", "|S1", "|S2", "|S3", "|S4", "|O"]:
Review comment:
probably worth checking out
https://github.com/dask/distributed/blob/master/distributed/protocol/tests/test_numpy.py#L30-L65
see https://issues.apache.org/jira/browse/ARROW-1596
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Handle more dtypes efficiently in custom numpy array serializer.
> ----------------------------------------------------------------
>
> Key: ARROW-2451
> URL: https://issues.apache.org/jira/browse/ARROW-2451
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
> Reporter: Robert Nishihara
> Assignee: Robert Nishihara
> Priority: Major
> Labels: pull-request-available
>
> Right now certain dtypes like bool or fixed length strings are serialized as
> lists, which is inefficient. We can handle these more efficiently by casting
> them to uint8 and saving the original dtype as additional data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
