[
https://issues.apache.org/jira/browse/ARROW-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361562#comment-16361562
]
ASF GitHub Bot commented on ARROW-1860:
---------------------------------------
wesm commented on issue #1500: ARROW-1860: [C++] Introduce ipc::PreparedMessage
data structure to avoid making multiple passes over record batches
URL: https://github.com/apache/arrow/pull/1500#issuecomment-365091190
So it seems that not all Python file objects support the `tell` operation,
for example the `_io.BufferedReader` that you get calling `makefile` on a
socket. That's the last issue here -- I think probably the best way to deal
with this is to implement a `PyInputStream` in
https://github.com/apache/arrow/blob/master/cpp/src/arrow/python/io.h and
handle the stream position bookkeeping internally, if that sounds good to folks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [C++] Add data structure to "stage" a sequence of IPC messages from in-memory
> data
> ----------------------------------------------------------------------------------
>
> Key: ARROW-1860
> URL: https://issues.apache.org/jira/browse/ARROW-1860
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Wes McKinney
> Assignee: Wes McKinney
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
> Attachments: text.html
>
>
> Currently, when you need to pre-allocate space for a record batch or a stream
> (schema + dictionaries + record batches), you must make multiple passes over
> the data structures of interest (and use e.g. {{MockOutputStream}} to compute
> the size of the output buffer). It would be useful to make a single pass to
> "prepare" the IPC payload for both sizing and writing to prevent having to
> make multiple passes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
