I tried to reproduce it and when I re-cloned my repo, it worked fine. There must have been some holdover file from when I switched from 0.8.0. Good note to self: always do that before bothering others!
Thank you for helping me debug. On Tue, Mar 27, 2018, 7:47 PM Wes McKinney <wesmck...@gmail.com> wrote: > > But, how do I access either the protected mutable_data_ or the inline > mutable_data from my this wrapper? > > I may be missing something, but you need to use Buffer::mutable_data() > somewhere in the wrapper. So either provide a function in the wrapper > that calls it, or invoke it during some init step (if you are copying > fields into a C struct) > > On Tue, Mar 27, 2018 at 4:33 PM, Philipp Moritz <pcmor...@gmail.com> > wrote: > > Also, if you provide me with full instructions on how to reproduce the > > linking error, I'm happy to look if I can figure out what is going on. > > > > On Tue, Mar 27, 2018 at 1:25 PM, Philipp Moritz <pcmor...@gmail.com> > wrote: > > > >> Hey Saul, > >> > >> The Buffer returned by Plasma's Create is not owning the memory, the > >> memory management happens via reference counting through the explicit > >> PlasmaClient::Release call. The reason why we now return a buffer > instead > >> of a plain pointer and length is so we can support both CPU > (MutableBuffer) > >> and GPU (CudaBuffer) memory with the same interface. > >> > >> If you only use CPU memory, it is safe to dynamically cast the Buffer > >> returned by Create to a MutableBuffer and get it's underlying pointer > via > >> the mutable_data() function. > >> > >> This behavior might change in the future however if we decide to do > >> reference counting through the Buffer returned by PlasmaClient::Create. > >> However if we are merging libplasma into arrow soon I think it is save > to > >> depend on the behavior as it is at the moment. > >> > >> -- Philipp. > >> > >> On Tue, Mar 27, 2018 at 1:04 PM, Saul Shanabrook < > s.shanabr...@gmail.com> > >> wrote: > >> > >>> Yeah, I am writing C++ code that provides a C wrapper around this. But, > >>> how > >>> do I access either the protected mutable_data_ or the inline > mutable_data > >>> from my this wrapper? > >>> > >>> On Tue, Mar 27, 2018 at 10:52 AM Wes McKinney <wesmck...@gmail.com> > >>> wrote: > >>> > >>> > @Kou, we removed that bit, so Buffer::mutable_data is always inline > now. > >>> > > >>> > @Saul, you will want to create a C wrapper for arrow::Buffer so you > >>> > can manage memory lifetime from C. Depending on the kind of Buffer > you > >>> > have, it may not be as simple as calling free(...) on its internal > >>> > data. > >>> > > >>> > - Wes > >>> > > >>> > On Tue, Mar 27, 2018 at 10:31 AM, Kouhei Sutou <k...@clear-code.com> > >>> wrote: > >>> > > Hi, > >>> > > > >>> > > It may be wrong but you may need to add -DNDEBUG flag to C > >>> > > compiler. > >>> > > > >>> > > > >>> > > Thanks, > >>> > > -- > >>> > > kou > >>> > > > >>> > > In <CANapY_LzahViR9e7e90H=wq3j0tnbZJe1Vm=j7=vvyqawnc...@mail.gm > >>> ail.com> > >>> > > "Re: Using Plasma with xnd" on Tue, 27 Mar 2018 13:41:49 +0000, > >>> > > Saul Shanabrook <s.shanabr...@gmail.com> wrote: > >>> > > > >>> > >> Ah, I see I can use garrow_buffer_get_mutable_data from c_glib. > >>> > >> > >>> > >> On Tue, Mar 27, 2018 at 9:29 AM Saul Shanabrook < > >>> s.shanabr...@gmail.com > >>> > > > >>> > >> wrote: > >>> > >> > >>> > >>> I am in the process of changing over my C wrapper to work with > Arrow > >>> > >>> 0.9.0. One change is that the plasma APIs now use arrow::Buffer > >>> > instead of > >>> > >>> of raw pointers, in Create and Get. > >>> > >>> > >>> > >>> In my C wrapper for Create, I need to get the mutable data > pointer > >>> from > >>> > >>> the arrow::Buffer, after it is created. Here is my current > >>> > implementation: > >>> > >>> > >>> > https://github.com/plures/pxnd/blob/84e9403ef3811e0893e12076 > >>> 455d7cbc49c39358/libplasma/plasma.cc#L56-L75 > >>> > >>> I do something similar to the Python wrapper around plasma: > >>> > >>> > >>> > https://github.com/apache/arrow/blob/f9f8320339692d4134d1ef4 > >>> 2c32cb7c8d547593e/python/pyarrow/plasma.pyx#L300-L307 > >>> > >>> > >>> > >>> > >>> > >>> However, this fails to link, I think because the mutable_data > >>> method is > >>> > >>> defined in the class and so is inline. So it doesn't show up in > the > >>> > >>> libarrow.dylib/libarrow.a archive. Let me know if I am missing > >>> > something > >>> > >>> obvious, this is just my hypothesis for why it isn't working. > >>> > >>> > >>> > >>> Is there a way for a third party library to get the pointer to > the > >>> data > >>> > >>> from a arrow::Buffer? > >>> > >>> > >>> > >>> On Wed, Mar 7, 2018 at 10:59 AM Wes McKinney < > wesmck...@gmail.com> > >>> > wrote: > >>> > >>> > >>> > >>>> Where are you going to make Plasma requests, in Python or in C? > >>> > >>>> > >>> > >>>> * If in Python, use the buffer protocol to access the memory in > C > >>> > >>>> https://docs.python.org/3/c-api/buffer.html#bufferobjects > >>> > >>>> * If in C, then I would suggest adding extern "C" APIs directly > in > >>> > >>>> libplasma.so so you can connect to the Plasma store and make > >>> requests > >>> > >>>> directly in C. > >>> > >>>> > >>> > >>>> I want to make 100% sure that you need to interact with the > Plasma > >>> > >>>> client in C before I send you down that path. Here's an example > of > >>> a C > >>> > >>>> API embedded in an otherwise C++ library > >>> > >>>> > >>> > > https://github.com/wesm/feather/blob/master/cpp/src/feather/feather-c.h > >>> . > >>> > >>>> > >>> > >>>> - Wes > >>> > >>>> > >>> > >>>> On Wed, Mar 7, 2018 at 10:53 AM, Saul Shanabrook < > >>> > s.shanabr...@gmail.com> > >>> > >>>> wrote: > >>> > >>>> > Hey Wes, > >>> > >>>> > > >>> > >>>> > Since all our logic for writing and reading the data is > written > >>> in > >>> > C, we > >>> > >>>> > need to be get access to the memory in C as well. I think it > >>> makes > >>> > sense > >>> > >>>> > then for me to work on creating a C++ wrapper around Plasma > that > >>> > exports > >>> > >>>> > what is necessary in C. Should that be in Arrow GLib? > >>> > >>>> > > >>> > >>>> > Best, > >>> > >>>> > Saul > >>> > >>>> > > >>> > >>>> > On Tue, Mar 6, 2018 at 11:49 PM Wes McKinney < > >>> wesmck...@gmail.com> > >>> > >>>> wrote: > >>> > >>>> > > >>> > >>>> >> hi Saul -- I think the easiest solution here is the > >>> > buffer/memoryview > >>> > >>>> >> protocol. You won't have to touch the Cython or C++ API from > >>> > pyarrow > >>> > >>>> >> if you do this. > >>> > >>>> >> > >>> > >>>> >> You can interact with a Buffer object like any other Python > >>> object > >>> > >>>> >> implementing the buffer protocol. See numpy.frombuffer as an > >>> > example > >>> > >>>> >> of a function that interacts with such objects. I would > suggest > >>> > adding > >>> > >>>> >> a method to xnd method for this. > >>> > >>>> >> > >>> > >>>> >> If you need to interact with Plasma from C then things will > be > >>> more > >>> > >>>> >> complicated -- Robert or Philipp should be able to advise in > >>> this > >>> > >>>> >> case. > >>> > >>>> >> > >>> > >>>> >> - Wes > >>> > >>>> >> > >>> > >>>> >> On Tue, Mar 6, 2018 at 4:55 PM, Saul Shanabrook < > >>> > >>>> s.shanabr...@gmail.com> > >>> > >>>> >> wrote: > >>> > >>>> >> > Hey Wes, > >>> > >>>> >> > > >>> > >>>> >> > I don't have much experience doing C + Python + Cython > >>> > development, > >>> > >>>> so I > >>> > >>>> >> am > >>> > >>>> >> > probably missing something obvious, but reading the Cython > >>> docs, > >>> > >>>> >> > < > >>> > >>>> >> > >>> > >>>> > >>> > https://cython.readthedocs.io/en/latest/src/userguide/extern > >>> al_C_code.html#public-declarations > >>> > >>>> >> > > >>> > >>>> >> > it > >>> > >>>> >> > seems like I can only access types marked as public from C > >>> code. > >>> > >>>> When I > >>> > >>>> >> > compile arrow locally, I do get some C++ headers for the > >>> plasma > >>> > code, > >>> > >>>> >> but I > >>> > >>>> >> > don't think I can use them from C code either. > >>> > >>>> >> > > >>> > >>>> >> > Best, > >>> > >>>> >> > Saul > >>> > >>>> >> > > >>> > >>>> >> > > >>> > >>>> >> > > >>> > >>>> >> > On Tue, Mar 6, 2018 at 3:12 PM Wes McKinney < > >>> wesmck...@gmail.com > >>> > > > >>> > >>>> wrote: > >>> > >>>> >> > > >>> > >>>> >> >> hi Saul, > >>> > >>>> >> >> > >>> > >>>> >> >> Are you able to use the buffer/memoryview protocol? > >>> Instances of > >>> > >>>> >> >> pyarrow.Buffer, like PlasmaBuffer, support this > >>> > >>>> >> >> > >>> > >>>> >> >> > >>> > >>>> >> > >>> > >>>> > >>> > https://github.com/apache/arrow/blob/master/python/pyarrow/p > >>> lasma.pyx#L182 > >>> > >>>> >> >> > >>> > >>>> >> >> - Wes > >>> > >>>> >> >> > >>> > >>>> >> >> On Tue, Mar 6, 2018 at 3:09 PM, Saul Shanabrook < > >>> > >>>> s.shanabr...@gmail.com > >>> > >>>> >> > > >>> > >>>> >> >> wrote: > >>> > >>>> >> >> > I am trying to use the Plasma store to back xnd objects. > >>> Xnd ( > >>> > >>>> >> >> > https://xnd.readthedocs.io/en/latest/xnd/index.html) > is a > >>> > >>>> container > >>> > >>>> >> >> library > >>> > >>>> >> >> > in C that has Python bindings. I would like to get a > >>> pointer > >>> > to > >>> > >>>> the > >>> > >>>> >> >> > allocated memory after creating or get an object in > >>> Plasma. I > >>> > see > >>> > >>>> that > >>> > >>>> >> >> this > >>> > >>>> >> >> > is supported in the C++ API ( > >>> > >>>> >> >> > > >>> > >>>> >> >> > >>> > >>>> >> > >>> > >>>> > >>> > https://arrow.apache.org/docs/cpp/classplasma_1_1_plasma_cli > >>> ent.html#ac18ab9cc792c620a97a3dcb165e0ecd7 > >>> > >>>> >> >> ) > >>> > >>>> >> >> > but not in the python API (as far as I can tell). Is it > >>> > possible > >>> > >>>> to > >>> > >>>> >> use > >>> > >>>> >> >> the > >>> > >>>> >> >> > C++ Plasma API from a C project? If not, would it make > >>> sense > >>> > to > >>> > >>>> expose > >>> > >>>> >> >> > pointer access on the Python API using capsules > >>> > >>>> >> >> > https://docs.python.org/3.6/c-api/capsule.html > >>> > >>>> >> >> > < > https://docs.python.org/3.6/c-api/capsule.html#capsules>? > >>> > >>>> >> >> > >>> > >>>> >> > >>> > >>>> > >>> > >>> > >>> > > >>> > >> > >> >
