I discussed some of these things at a high level in my talk at SciPy 2018 last week
https://www.slideshare.net/wesm/apache-arrow-crosslanguage-development-platform-for-inmemory-data-105427919 On Mon, Jul 16, 2018 at 2:08 PM, Wes McKinney <wesmck...@gmail.com> wrote: > hi Yurui, > > You can also share data structures through JNI without using the IPC > tools at all, which could require memory copying to produce the IPC > messages. > > What you can do is obtain the memory addresses for the component > buffers of an array (or vector, as called in Java) and construct the > data structure from the memory addresses on the other side. We are > doing exactly this already in Python using JPype (which is JNI-based): > > https://github.com/apache/arrow/blob/master/python/pyarrow/jvm.py > > The Gandiva project uses JNI to pass Java Netty buffer memory > addresses to C++, you can see the code for creating the arrays from > the memory addresses and then constructing a RecordBatch: > > https://github.com/dremio/gandiva/blob/master/cpp/src/jni/native_builder.cc#L602 > > I believe as time goes on we will have better and more standardized > APIs to deal with JNI<->C++ zero-copy passing, these implementations > have only been done relatively recently. Your contributions to the > Arrow project around this would be most welcomed! > > Thanks, > Wes > > On Mon, Jul 16, 2018 at 2:00 PM, Philipp Moritz <pcmor...@gmail.com> wrote: >> Hey Yuri, >> >> you can use the Arrow IPC mechanism to do this: >> >> - https://github.com/apache/arrow/blob/master/format/IPC.md >> - Python: https://arrow.apache.org/docs/python/ipc.html >> - C++: https://arrow.apache.org/docs/cpp/namespacearrow_1_1ipc.html >> - For Java, see the org.apache.arrow.vector.ipc namespace >> >> On the C++ side, you can for example use a RecordBatchStreamWriter to write >> the IPC message, and then on the Java side you could use the >> ArrowStreamReader to read it. >> >> There are some tests here: >> https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/ipc-read-write-test.cc >> https://github.com/apache/arrow/tree/master/java/vector/src/test/java/org/apache/arrow/vector/ipc >> >> There is also integration tests here, although I'm not really familiar with >> them: >> >> https://github.com/apache/arrow/tree/master/integration >> >> If you could write a little tutorial/into on how to do this (maybe using >> Plasma for exchanging the data) and contribute it to the documentation, >> that would be amazing! >> >> Best, >> Philipp. >> >> On Mon, Jul 16, 2018 at 4:14 AM, 周宇睿(闻拙) <yurui....@alibaba-inc.com> wrote: >> >>> Hi guys: >>> >>> I might miss something quite obviously. But how does Arrow passing objects >>> across language? Let’s say I have a java program that invoke a c++ function >>> via JNI, how does the c++ function pass an Arrow RecordBack object back to >>> Java runtime without memory copy? >>> >>> Any advise would be appreciated. >>> Thanks >>> Yurui >>> >>> from Alimail macOS
