hi Rares -- IPC messages produced by 0.9.0 should be forward compatible. I opened https://issues.apache.org/jira/browse/ARROW-921 some time ago about adding some tools to integration test one version versus another to obtain hard proof of this, but this work has not been completed yet (any takers?).
Have you encountered any problems? Thanks, Wes On Sun, Mar 10, 2019 at 11:49 PM Rares Vernica <rvern...@gmail.com> wrote: > > Hello, > > I have a C++ library using Arrow 0.9.0 to serialize data The code looks > like this: > > std::shared_ptr<arrow::RecordBatch> arrowBatch; > arrowBatch = arrow::RecordBatch::Make(_arrowSchema, nCells, _arrowArrays); > > std::shared_ptr<arrow::PoolBuffer> arrowBuffer(new > arrow::PoolBuffer(_arrowPool)); > arrow::io::BufferOutputStream arrowStream(arrowBuffer); > > std::shared_ptr<arrow::ipc::RecordBatchWriter> arrowWriter; > arrow::ipc::RecordBatchStreamWriter::Open(&arrowStream, _arrowSchema, > &arrowWriter); > > arrowWriter->WriteRecordBatch(*arrowBatch); > ... > reinterpret_cast<const char*>(arrowBuffer->data()), arrowBuffer->size()) > ... > > The output bytes are then read in Python using pyarrow: > > pyarrow.RecordBatchStreamReader(pyarrow.BufferReader(buf)).read_pandas() > > Since the C++ side uses Arrow 0.9.0 I have been using pyarrow==0.9.0. When > using Python 3.7, getting pyarrow=0.9.0 is not easy since there are no > per-compiled .whl packages on PyPI. > > I wonder if I could use newer pyarrow versions to parse the Arrow 0.9.0 > ouput? Is the format compatible? > > Thanks! > Rares
