[
https://issues.apache.org/jira/browse/ARROW-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16432016#comment-16432016
]
ASF GitHub Bot commented on ARROW-1860:
---------------------------------------
pitrou commented on issue #1500: ARROW-1860: [C++] Introduce
ipc::PreparedMessage data structure to avoid making multiple passes over record
batches
URL: https://github.com/apache/arrow/pull/1500#issuecomment-380045557
> So it seems that not all Python file objects support the tell operation,
for example the _io.BufferedReader that you get calling makefile on a socket.
Yes, non-seekable files generally don't support tell() either.
I'm a bit surprised that you need this: if writing to a generic output
stream, why do you want to pad data? Padding only sounds useful when writing to
a true random access file that may be e.g. memory-mapped.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> [C++] Add data structure to "stage" a sequence of IPC messages from in-memory
> data
> ----------------------------------------------------------------------------------
>
> Key: ARROW-1860
> URL: https://issues.apache.org/jira/browse/ARROW-1860
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Wes McKinney
> Assignee: Wes McKinney
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
> Attachments: text.html
>
>
> Currently, when you need to pre-allocate space for a record batch or a stream
> (schema + dictionaries + record batches), you must make multiple passes over
> the data structures of interest (and use e.g. {{MockOutputStream}} to compute
> the size of the output buffer). It would be useful to make a single pass to
> "prepare" the IPC payload for both sizing and writing to prevent having to
> make multiple passes
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
