[
https://issues.apache.org/jira/browse/ARROW-474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15818747#comment-15818747
]
Wes McKinney edited comment on ARROW-474 at 1/11/17 4:17 PM:
-------------------------------------------------------------
+1, sounds good to me. This won't permit random access of batches, so not a
replacement for the current file format, but very useful for streaming use
cases as you describe. We have all the tools to implement this both in Java and
C++ without a great of effort -- if we formalize the streaming format in the
spec documents, then we can create initial implementations. We obviously will
want to write integration tests for it.
was (Author: wesmckinn):
Sounds good to me. This won't permit random access of batches, so not a
replacement for the current file format, but very useful for streaming use
cases as you describe. We have all the tools to implement this both in Java and
C++ without a great of effort -- if we formalize the streaming format in the
spec documents, then we can create initial implementations. We obviously will
want to write integration tests for it.
> Create an Arrow streaming file fomat
> ------------------------------------
>
> Key: ARROW-474
> URL: https://issues.apache.org/jira/browse/ARROW-474
> Project: Apache Arrow
> Issue Type: Improvement
> Reporter: Nong Li
>
> It would be helpful to have a file/container layout that allows for streaming
> consumer and producers of arrow data.
> As a motivating example, a reader could have this API:
> Iterator<Record> read(java.io.InputStream source).
> Similar a writer can have:
> void write(RecordBatch, java.io.OutputStream dest)
> The streams can be from a variety of sources: it be files, over the network,
> shared memory, pipe, etc. The layout would make it possible to implement
> both APIs without requiring intermediate buffering.
> Speaking with Julien, this involves creating a simple container format
> similar to ARROW-264 but suitable for streaming consumer and producers. The
> layout in ARROW-264 has the schema at the end so can't be used in a streaming
> way. This format would likely be a rearranging of the current File layout
> pieces, something similar to:
> MAGIC, schema, totalBatches, (RecordBatchHeader, RecordBatchBody)*, MAGIC (or
> some end marker)
> totalBatches is optional for streaming producers that don't know.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
