[
https://issues.apache.org/jira/browse/ARROW-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16266155#comment-16266155
]
ASF GitHub Bot commented on ARROW-1783:
---------------------------------------
mrocklin commented on issue #1362: ARROW-1783: [Python] Provide a "component"
dict representation of a serialized Python object with minimal allocation
URL: https://github.com/apache/arrow/pull/1362#issuecomment-347033537
At first glance the `to_components`/`deserialize_components` structure seems
good to me. This is definitely something that we can work with on the Dask
side. Is it possible to easily turn each of the elements of
`to_components()['data']` into a `memoryview` without significant cost?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> [Python] Convert SerializedPyObject to/from sequence of component buffers
> with minimal memory allocation / copying
> ------------------------------------------------------------------------------------------------------------------
>
> Key: ARROW-1783
> URL: https://issues.apache.org/jira/browse/ARROW-1783
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Python
> Reporter: Wes McKinney
> Assignee: Wes McKinney
> Labels: pull-request-available
> Fix For: 0.8.0
>
>
> See discussion on Dask org:
> https://github.com/dask/distributed/pull/931
> It would be valuable for downstream users to compute the serialized payload
> as a sequence of memoryview-compatible objects without having to allocate new
> memory on write. This means that the component tensor messages must have
> their metadata and bodies in separate buffers. This will require a bit of
> work internally reassemble the object from a collection of {{pyarrow.Buffer}}
> objects
> see also ARROW-1509
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
