[
https://issues.apache.org/jira/browse/ARROW-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477859#comment-16477859
]
Rob Ambalu commented on ARROW-2579:
-----------------------------------
So I was just about to write a simple repro but I realize I cant because the
append mode patch for FileOutputStream hasnt been merged yet.
You need a streamer with an "append" option to repro the issue. As far as I
can see thers only FileOutputStream ( broken ) and hadoop, which im not sure
how to bootstrap. I hit this issue in my code because I wrote my own
MMapStreamer that supports append.
Ive actually worked around this at this point by creating new .idx extension
files if I already find an existing file and want to append ( reader then
concats them all )
> Appending to streamable table file format doesn't seem to work
> --------------------------------------------------------------
>
> Key: ARROW-2579
> URL: https://issues.apache.org/jira/browse/ARROW-2579
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Affects Versions: 0.9.0
> Reporter: Rob Ambalu
> Priority: Major
>
> As far as I can tell it looks like appending to a streaming file format isn’t
> currently supported, is that right?
> RecordBatchStreamWriter always writes the schema up front, and it doesn’t
> look like a schema is expected mid file ( assuming im doing this append test
> correctly, this is the error I hit when I try to read back this file into
> python:
> Traceback (most recent call last):
> File "/home/ra7293/rba_arrow_mmap.py", line 9, in <module>
> table = reader.read_all()
> File "ipc.pxi", line 302, in pyarrow.lib._RecordBatchReader.read_all
> File "error.pxi", line 79, in pyarrow.lib.check_status
> pyarrow.lib.ArrowIOError: Message not expected type: record batch, was: 1
>
> This reader script works fine if I write once / don’t append.
> Seeing as IO interfaces support Append, streaming should support it as well (
> if for whatever reason this cant be supported, RecordBatchStreamWriter should
> throw if configured with an OutputStreamer that is attempting to append )
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)