Hi Jacob, The current rough prototype/proposal of the IPC via shared memory is to do a depth first traversal of each arrays buffer and write them out to a contiguous memory block. Metadata about array types and locations of buffers is persisted at the end of memory block in flatbuffer format [1]. Reading it back is a matter of using the metadata to create a structure (like the one you have above) that has pointers back to the contiguous memory block. The work in progress C++ version of this located at [2].
I hope this helps. [1] https://github.com/apache/arrow/blob/master/format/Message.fbs [2] https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/adapter.cc Cheers, Micah On Wed, Jun 1, 2016 at 10:53 AM, Jacob Quinn <quinn.jac...@gmail.com> wrote: > Having become familiar with the Arrow memory layout, and taking a stab at > an implementation in the Julia language, I've come up with a perhaps naive > question. > > A "type" (class) I have defined so far is: > > immutable Column{T} <: ArrowColumn{T} > buffer::Vector{UInt8} # potential reference to mmap > length::Int32 > null_count::Int32 > nulls::BitVector # null == 0 == false, not-null == 1 == true; always > padded to 64-byte alignments > values::Vector{T} # always padded to 64-byte alignments > end > > > which aims to be an array/column that holds any "primitive" bits type `T`. > Note the exact layout matching with "length", "null_count", "nulls", and > "values". > > The additional reference, however, is the "buffer" field, which holds a > reference to a byte buffer. This would be technically optional if the > `nulls` and `values` fields owned their own memory, but there are other > cases where `buffer` would own, for example, memory-mapped bytes that > `nulls` and `values` would be sharing. > > My question is if this somehow "violates" the Arrow memory layout by > including this additional `buffer` reference in my class? > > It begs a larger question of what exactly the inter-language "API" looks > like. I'm assuming it's not as strict as needing to be able to pass a > pointer to another process that would be able to auto-wrap as it's own > Arrow structure; but I think I read somewhere that it IS aiming for some > kind of "memcpy" operation. Any light anyone can shed would be most > welcome; help me know if I'm perhaps over-thinking this at this stage. > > -Jacob
