Re: Using Plasma with xnd

[Saul Shanabrook]

I tried to reproduce it and when I re-cloned my repo, it worked fine. There
must have been some holdover file from when I switched from 0.8.0. Good
note to self: always do that before bothering others!

Thank you for helping me debug.

On Tue, Mar 27, 2018, 7:47 PM Wes McKinney  wrote:

> > But, how do I access either the protected mutable_data_  or the inline
> mutable_data from my this wrapper?
>
> I may be missing something, but you need to use Buffer::mutable_data()
> somewhere in the wrapper. So either provide a function in the wrapper
> that calls it, or invoke it during some init step (if you are copying
> fields into a C struct)
>
> On Tue, Mar 27, 2018 at 4:33 PM, Philipp Moritz 
> wrote:
> > Also, if you provide me with full instructions on how to reproduce the
> > linking error, I'm happy to look if I can figure out what is going on.
> >
> > On Tue, Mar 27, 2018 at 1:25 PM, Philipp Moritz 
> wrote:
> >
> >> Hey Saul,
> >>
> >> The Buffer returned by Plasma's Create is not owning the memory, the
> >> memory management happens via reference counting through the explicit
> >> PlasmaClient::Release call. The reason why we now return a buffer
> instead
> >> of a plain pointer and length is so we can support both CPU
> (MutableBuffer)
> >> and GPU (CudaBuffer) memory with the same interface.
> >>
> >> If you only use CPU memory, it is safe to dynamically cast the Buffer
> >> returned by Create to a MutableBuffer and get it's underlying pointer
> via
> >> the mutable_data() function.
> >>
> >> This behavior might change in the future however if we decide to do
> >> reference counting through the Buffer returned by PlasmaClient::Create.
> >> However if we are merging libplasma into arrow soon I think it is save
> to
> >> depend on the behavior as it is at the moment.
> >>
> >> -- Philipp.
> >>
> >> On Tue, Mar 27, 2018 at 1:04 PM, Saul Shanabrook <
> s.shanabr...@gmail.com>
> >> wrote:
> >>
> >>> Yeah, I am writing C++ code that provides a C wrapper around this. But,
> >>> how
> >>> do I access either the protected mutable_data_  or the inline
> mutable_data
> >>> from my this wrapper?
> >>>
> >>> On Tue, Mar 27, 2018 at 10:52 AM Wes McKinney 
> >>> wrote:
> >>>
> >>> > @Kou, we removed that bit, so Buffer::mutable_data is always inline
> now.
> >>> >
> >>> > @Saul, you will want to create a C wrapper for arrow::Buffer so you
> >>> > can manage memory lifetime from C. Depending on the kind of Buffer
> you
> >>> > have, it may not be as simple as calling free(...) on its internal
> >>> > data.
> >>> >
> >>> > - Wes
> >>> >
> >>> > On Tue, Mar 27, 2018 at 10:31 AM, Kouhei Sutou 
> >>> wrote:
> >>> > > Hi,
> >>> > >
> >>> > > It may be wrong but you may need to add -DNDEBUG flag to C
> >>> > > compiler.
> >>> > >
> >>> > >
> >>> > > Thanks,
> >>> > > --
> >>> > > kou
> >>> > >
> >>> > > In 

Re: Using Plasma with xnd

[Philipp Moritz]

Also, if you provide me with full instructions on how to reproduce the
linking error, I'm happy to look if I can figure out what is going on.

On Tue, Mar 27, 2018 at 1:25 PM, Philipp Moritz  wrote:

> Hey Saul,
>
> The Buffer returned by Plasma's Create is not owning the memory, the
> memory management happens via reference counting through the explicit
> PlasmaClient::Release call. The reason why we now return a buffer instead
> of a plain pointer and length is so we can support both CPU (MutableBuffer)
> and GPU (CudaBuffer) memory with the same interface.
>
> If you only use CPU memory, it is safe to dynamically cast the Buffer
> returned by Create to a MutableBuffer and get it's underlying pointer via
> the mutable_data() function.
>
> This behavior might change in the future however if we decide to do
> reference counting through the Buffer returned by PlasmaClient::Create.
> However if we are merging libplasma into arrow soon I think it is save to
> depend on the behavior as it is at the moment.
>
> -- Philipp.
>
> On Tue, Mar 27, 2018 at 1:04 PM, Saul Shanabrook 
> wrote:
>
>> Yeah, I am writing C++ code that provides a C wrapper around this. But,
>> how
>> do I access either the protected mutable_data_  or the inline mutable_data
>> from my this wrapper?
>>
>> On Tue, Mar 27, 2018 at 10:52 AM Wes McKinney 
>> wrote:
>>
>> > @Kou, we removed that bit, so Buffer::mutable_data is always inline now.
>> >
>> > @Saul, you will want to create a C wrapper for arrow::Buffer so you
>> > can manage memory lifetime from C. Depending on the kind of Buffer you
>> > have, it may not be as simple as calling free(...) on its internal
>> > data.
>> >
>> > - Wes
>> >
>> > On Tue, Mar 27, 2018 at 10:31 AM, Kouhei Sutou 
>> wrote:
>> > > Hi,
>> > >
>> > > It may be wrong but you may need to add -DNDEBUG flag to C
>> > > compiler.
>> > >
>> > >
>> > > Thanks,
>> > > --
>> > > kou
>> > >
>> > > In 

Re: Using Plasma with xnd

[Philipp Moritz]

Hey Saul,

The Buffer returned by Plasma's Create is not owning the memory, the memory
management happens via reference counting through the explicit
PlasmaClient::Release call. The reason why we now return a buffer instead
of a plain pointer and length is so we can support both CPU (MutableBuffer)
and GPU (CudaBuffer) memory with the same interface.

If you only use CPU memory, it is safe to dynamically cast the Buffer
returned by Create to a MutableBuffer and get it's underlying pointer via
the mutable_data() function.

This behavior might change in the future however if we decide to do
reference counting through the Buffer returned by PlasmaClient::Create.
However if we are merging libplasma into arrow soon I think it is save to
depend on the behavior as it is at the moment.

-- Philipp.

On Tue, Mar 27, 2018 at 1:04 PM, Saul Shanabrook 
wrote:

> Yeah, I am writing C++ code that provides a C wrapper around this. But, how
> do I access either the protected mutable_data_  or the inline mutable_data
> from my this wrapper?
>
> On Tue, Mar 27, 2018 at 10:52 AM Wes McKinney  wrote:
>
> > @Kou, we removed that bit, so Buffer::mutable_data is always inline now.
> >
> > @Saul, you will want to create a C wrapper for arrow::Buffer so you
> > can manage memory lifetime from C. Depending on the kind of Buffer you
> > have, it may not be as simple as calling free(...) on its internal
> > data.
> >
> > - Wes
> >
> > On Tue, Mar 27, 2018 at 10:31 AM, Kouhei Sutou 
> wrote:
> > > Hi,
> > >
> > > It may be wrong but you may need to add -DNDEBUG flag to C
> > > compiler.
> > >
> > >
> > > Thanks,
> > > --
> > > kou
> > >
> > > In 

Re: Using Plasma with xnd

[Saul Shanabrook]

Yeah, I am writing C++ code that provides a C wrapper around this. But, how
do I access either the protected mutable_data_  or the inline mutable_data
from my this wrapper?

On Tue, Mar 27, 2018 at 10:52 AM Wes McKinney  wrote:

> @Kou, we removed that bit, so Buffer::mutable_data is always inline now.
>
> @Saul, you will want to create a C wrapper for arrow::Buffer so you
> can manage memory lifetime from C. Depending on the kind of Buffer you
> have, it may not be as simple as calling free(...) on its internal
> data.
>
> - Wes
>
> On Tue, Mar 27, 2018 at 10:31 AM, Kouhei Sutou  wrote:
> > Hi,
> >
> > It may be wrong but you may need to add -DNDEBUG flag to C
> > compiler.
> >
> >
> > Thanks,
> > --
> > kou
> >
> > In 

Re: Using Plasma with xnd

[Wes McKinney]

@Kou, we removed that bit, so Buffer::mutable_data is always inline now.

@Saul, you will want to create a C wrapper for arrow::Buffer so you
can manage memory lifetime from C. Depending on the kind of Buffer you
have, it may not be as simple as calling free(...) on its internal
data.

- Wes

On Tue, Mar 27, 2018 at 10:31 AM, Kouhei Sutou  wrote:
> Hi,
>
> It may be wrong but you may need to add -DNDEBUG flag to C
> compiler.
>
>
> Thanks,
> --
> kou
>
> In 

Re: Using Plasma with xnd

[Kouhei Sutou]

Hi,

It may be wrong but you may need to add -DNDEBUG flag to C
compiler.


Thanks,
--
kou

In 

Re: Using Plasma with xnd

[Saul Shanabrook]

Ah, I see I can use garrow_buffer_get_mutable_data from c_glib.

On Tue, Mar 27, 2018 at 9:29 AM Saul Shanabrook 
wrote:

> I am in the process of changing over my C wrapper to work with Arrow
> 0.9.0. One change is that the plasma APIs now use arrow::Buffer instead of
> of raw pointers, in Create and Get.
>
> In my C wrapper for Create, I need to get the mutable data pointer from
> the arrow::Buffer, after it is created. Here is my current implementation:
> https://github.com/plures/pxnd/blob/84e9403ef3811e0893e12076455d7cbc49c39358/libplasma/plasma.cc#L56-L75
> I do something similar to the Python wrapper around plasma:
> https://github.com/apache/arrow/blob/f9f8320339692d4134d1ef42c32cb7c8d547593e/python/pyarrow/plasma.pyx#L300-L307
>
>
> However, this fails to link, I think because the mutable_data method is
> defined in the class and so is inline. So it doesn't show up in the
> libarrow.dylib/libarrow.a archive. Let me know if I am missing something
> obvious, this is just my hypothesis for why it isn't working.
>
> Is there a way for a third party library to get the pointer to the data
> from a arrow::Buffer?
>
> On Wed, Mar 7, 2018 at 10:59 AM Wes McKinney  wrote:
>
>> Where are you going to make Plasma requests, in Python or in C?
>>
>> * If in Python, use the buffer protocol to access the memory in C
>> https://docs.python.org/3/c-api/buffer.html#bufferobjects
>> * If in C, then I would suggest adding extern "C" APIs directly in
>> libplasma.so so you can connect to the Plasma store and make requests
>> directly in C.
>>
>> I want to make 100% sure that you need to interact with the Plasma
>> client in C before I send you down that path. Here's an example of a C
>> API embedded in an otherwise C++ library
>> https://github.com/wesm/feather/blob/master/cpp/src/feather/feather-c.h.
>>
>> - Wes
>>
>> On Wed, Mar 7, 2018 at 10:53 AM, Saul Shanabrook 
>> wrote:
>> > Hey Wes,
>> >
>> > Since all our logic for writing and reading the data is written in C, we
>> > need to be get access to the memory in C as well. I think it makes sense
>> > then for me to work on creating a C++ wrapper around Plasma that exports
>> > what is necessary in C. Should that be in Arrow GLib?
>> >
>> > Best,
>> > Saul
>> >
>> > On Tue, Mar 6, 2018 at 11:49 PM Wes McKinney 
>> wrote:
>> >
>> >> hi Saul -- I think the easiest solution here is the buffer/memoryview
>> >> protocol. You won't have to touch the Cython or C++ API from pyarrow
>> >> if you do this.
>> >>
>> >> You can interact with a Buffer object like any other Python object
>> >> implementing the buffer protocol. See numpy.frombuffer as an example
>> >> of a function that interacts with such objects. I would suggest adding
>> >> a method to xnd method for this.
>> >>
>> >> If you need to interact with Plasma from C then things will be more
>> >> complicated -- Robert or Philipp should be able to advise in this
>> >> case.
>> >>
>> >> - Wes
>> >>
>> >> On Tue, Mar 6, 2018 at 4:55 PM, Saul Shanabrook <
>> s.shanabr...@gmail.com>
>> >> wrote:
>> >> > Hey Wes,
>> >> >
>> >> > I don't have much experience doing C + Python + Cython development,
>> so I
>> >> am
>> >> > probably missing something obvious, but reading the Cython docs,
>> >> > <
>> >>
>> https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#public-declarations
>> >> >
>> >> > it
>> >> > seems like I can only access types marked as public from C code.
>> When I
>> >> > compile arrow locally, I do get some C++ headers for the plasma code,
>> >> but I
>> >> > don't think I can use them from C code either.
>> >> >
>> >> > Best,
>> >> > Saul
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Mar 6, 2018 at 3:12 PM Wes McKinney 
>> wrote:
>> >> >
>> >> >> hi Saul,
>> >> >>
>> >> >> Are you able to use the buffer/memoryview protocol? Instances of
>> >> >> pyarrow.Buffer, like PlasmaBuffer, support this
>> >> >>
>> >> >>
>> >>
>> https://github.com/apache/arrow/blob/master/python/pyarrow/plasma.pyx#L182
>> >> >>
>> >> >> - Wes
>> >> >>
>> >> >> On Tue, Mar 6, 2018 at 3:09 PM, Saul Shanabrook <
>> s.shanabr...@gmail.com
>> >> >
>> >> >> wrote:
>> >> >> > I am trying to use the Plasma store to back xnd objects. Xnd (
>> >> >> > https://xnd.readthedocs.io/en/latest/xnd/index.html) is a
>> container
>> >> >> library
>> >> >> > in C that has Python bindings. I would like to get a pointer to
>> the
>> >> >> > allocated memory after creating or get an object in Plasma. I see
>> that
>> >> >> this
>> >> >> > is supported in the C++ API (
>> >> >> >
>> >> >>
>> >>
>> https://arrow.apache.org/docs/cpp/classplasma_1_1_plasma_client.html#ac18ab9cc792c620a97a3dcb165e0ecd7
>> >> >> )
>> >> >> > but not in the python API (as far as I can tell). Is it possible
>> to
>> >> use
>> >> >> the
>> >> >> > C++ Plasma API from a C project? If not, would it make sense to
>> expose
>> >> >> > pointer access on the Python 

Re: Using Plasma with xnd

[Saul Shanabrook]

I am in the process of changing over my C wrapper to work with Arrow 0.9.0.
One change is that the plasma APIs now use arrow::Buffer instead of of raw
pointers, in Create and Get.

In my C wrapper for Create, I need to get the mutable data pointer from the
arrow::Buffer, after it is created. Here is my current implementation:
https://github.com/plures/pxnd/blob/84e9403ef3811e0893e12076455d7cbc49c39358/libplasma/plasma.cc#L56-L75
I do something similar to the Python wrapper around plasma:
https://github.com/apache/arrow/blob/f9f8320339692d4134d1ef42c32cb7c8d547593e/python/pyarrow/plasma.pyx#L300-L307


However, this fails to link, I think because the mutable_data method is
defined in the class and so is inline. So it doesn't show up in the
libarrow.dylib/libarrow.a archive. Let me know if I am missing something
obvious, this is just my hypothesis for why it isn't working.

Is there a way for a third party library to get the pointer to the data
from a arrow::Buffer?

On Wed, Mar 7, 2018 at 10:59 AM Wes McKinney  wrote:

> Where are you going to make Plasma requests, in Python or in C?
>
> * If in Python, use the buffer protocol to access the memory in C
> https://docs.python.org/3/c-api/buffer.html#bufferobjects
> * If in C, then I would suggest adding extern "C" APIs directly in
> libplasma.so so you can connect to the Plasma store and make requests
> directly in C.
>
> I want to make 100% sure that you need to interact with the Plasma
> client in C before I send you down that path. Here's an example of a C
> API embedded in an otherwise C++ library
> https://github.com/wesm/feather/blob/master/cpp/src/feather/feather-c.h.
>
> - Wes
>
> On Wed, Mar 7, 2018 at 10:53 AM, Saul Shanabrook 
> wrote:
> > Hey Wes,
> >
> > Since all our logic for writing and reading the data is written in C, we
> > need to be get access to the memory in C as well. I think it makes sense
> > then for me to work on creating a C++ wrapper around Plasma that exports
> > what is necessary in C. Should that be in Arrow GLib?
> >
> > Best,
> > Saul
> >
> > On Tue, Mar 6, 2018 at 11:49 PM Wes McKinney 
> wrote:
> >
> >> hi Saul -- I think the easiest solution here is the buffer/memoryview
> >> protocol. You won't have to touch the Cython or C++ API from pyarrow
> >> if you do this.
> >>
> >> You can interact with a Buffer object like any other Python object
> >> implementing the buffer protocol. See numpy.frombuffer as an example
> >> of a function that interacts with such objects. I would suggest adding
> >> a method to xnd method for this.
> >>
> >> If you need to interact with Plasma from C then things will be more
> >> complicated -- Robert or Philipp should be able to advise in this
> >> case.
> >>
> >> - Wes
> >>
> >> On Tue, Mar 6, 2018 at 4:55 PM, Saul Shanabrook  >
> >> wrote:
> >> > Hey Wes,
> >> >
> >> > I don't have much experience doing C + Python + Cython development,
> so I
> >> am
> >> > probably missing something obvious, but reading the Cython docs,
> >> > <
> >>
> https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#public-declarations
> >> >
> >> > it
> >> > seems like I can only access types marked as public from C code. When
> I
> >> > compile arrow locally, I do get some C++ headers for the plasma code,
> >> but I
> >> > don't think I can use them from C code either.
> >> >
> >> > Best,
> >> > Saul
> >> >
> >> >
> >> >
> >> > On Tue, Mar 6, 2018 at 3:12 PM Wes McKinney 
> wrote:
> >> >
> >> >> hi Saul,
> >> >>
> >> >> Are you able to use the buffer/memoryview protocol? Instances of
> >> >> pyarrow.Buffer, like PlasmaBuffer, support this
> >> >>
> >> >>
> >>
> https://github.com/apache/arrow/blob/master/python/pyarrow/plasma.pyx#L182
> >> >>
> >> >> - Wes
> >> >>
> >> >> On Tue, Mar 6, 2018 at 3:09 PM, Saul Shanabrook <
> s.shanabr...@gmail.com
> >> >
> >> >> wrote:
> >> >> > I am trying to use the Plasma store to back xnd objects. Xnd (
> >> >> > https://xnd.readthedocs.io/en/latest/xnd/index.html) is a
> container
> >> >> library
> >> >> > in C that has Python bindings. I would like to get a pointer to the
> >> >> > allocated memory after creating or get an object in Plasma. I see
> that
> >> >> this
> >> >> > is supported in the C++ API (
> >> >> >
> >> >>
> >>
> https://arrow.apache.org/docs/cpp/classplasma_1_1_plasma_client.html#ac18ab9cc792c620a97a3dcb165e0ecd7
> >> >> )
> >> >> > but not in the python API (as far as I can tell). Is it possible to
> >> use
> >> >> the
> >> >> > C++ Plasma API from a C project? If not, would it make sense to
> expose
> >> >> > pointer access on the Python API using capsules
> >> >> > https://docs.python.org/3.6/c-api/capsule.html
> >> >> > ?
> >> >>
> >>
>

Re: Using Plasma with xnd

[Wes McKinney]

Where are you going to make Plasma requests, in Python or in C?

* If in Python, use the buffer protocol to access the memory in C
https://docs.python.org/3/c-api/buffer.html#bufferobjects
* If in C, then I would suggest adding extern "C" APIs directly in
libplasma.so so you can connect to the Plasma store and make requests
directly in C.

I want to make 100% sure that you need to interact with the Plasma
client in C before I send you down that path. Here's an example of a C
API embedded in an otherwise C++ library
https://github.com/wesm/feather/blob/master/cpp/src/feather/feather-c.h.

- Wes

On Wed, Mar 7, 2018 at 10:53 AM, Saul Shanabrook  wrote:
> Hey Wes,
>
> Since all our logic for writing and reading the data is written in C, we
> need to be get access to the memory in C as well. I think it makes sense
> then for me to work on creating a C++ wrapper around Plasma that exports
> what is necessary in C. Should that be in Arrow GLib?
>
> Best,
> Saul
>
> On Tue, Mar 6, 2018 at 11:49 PM Wes McKinney  wrote:
>
>> hi Saul -- I think the easiest solution here is the buffer/memoryview
>> protocol. You won't have to touch the Cython or C++ API from pyarrow
>> if you do this.
>>
>> You can interact with a Buffer object like any other Python object
>> implementing the buffer protocol. See numpy.frombuffer as an example
>> of a function that interacts with such objects. I would suggest adding
>> a method to xnd method for this.
>>
>> If you need to interact with Plasma from C then things will be more
>> complicated -- Robert or Philipp should be able to advise in this
>> case.
>>
>> - Wes
>>
>> On Tue, Mar 6, 2018 at 4:55 PM, Saul Shanabrook 
>> wrote:
>> > Hey Wes,
>> >
>> > I don't have much experience doing C + Python + Cython development, so I
>> am
>> > probably missing something obvious, but reading the Cython docs,
>> > <
>> https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#public-declarations
>> >
>> > it
>> > seems like I can only access types marked as public from C code. When I
>> > compile arrow locally, I do get some C++ headers for the plasma code,
>> but I
>> > don't think I can use them from C code either.
>> >
>> > Best,
>> > Saul
>> >
>> >
>> >
>> > On Tue, Mar 6, 2018 at 3:12 PM Wes McKinney  wrote:
>> >
>> >> hi Saul,
>> >>
>> >> Are you able to use the buffer/memoryview protocol? Instances of
>> >> pyarrow.Buffer, like PlasmaBuffer, support this
>> >>
>> >>
>> https://github.com/apache/arrow/blob/master/python/pyarrow/plasma.pyx#L182
>> >>
>> >> - Wes
>> >>
>> >> On Tue, Mar 6, 2018 at 3:09 PM, Saul Shanabrook > >
>> >> wrote:
>> >> > I am trying to use the Plasma store to back xnd objects. Xnd (
>> >> > https://xnd.readthedocs.io/en/latest/xnd/index.html) is a container
>> >> library
>> >> > in C that has Python bindings. I would like to get a pointer to the
>> >> > allocated memory after creating or get an object in Plasma. I see that
>> >> this
>> >> > is supported in the C++ API (
>> >> >
>> >>
>> https://arrow.apache.org/docs/cpp/classplasma_1_1_plasma_client.html#ac18ab9cc792c620a97a3dcb165e0ecd7
>> >> )
>> >> > but not in the python API (as far as I can tell). Is it possible to
>> use
>> >> the
>> >> > C++ Plasma API from a C project? If not, would it make sense to expose
>> >> > pointer access on the Python API using capsules
>> >> > https://docs.python.org/3.6/c-api/capsule.html
>> >> > ?
>> >>
>>

Re: Using Plasma with xnd

[Saul Shanabrook]

Hey Wes,

Since all our logic for writing and reading the data is written in C, we
need to be get access to the memory in C as well. I think it makes sense
then for me to work on creating a C++ wrapper around Plasma that exports
what is necessary in C. Should that be in Arrow GLib?

Best,
Saul

On Tue, Mar 6, 2018 at 11:49 PM Wes McKinney  wrote:

> hi Saul -- I think the easiest solution here is the buffer/memoryview
> protocol. You won't have to touch the Cython or C++ API from pyarrow
> if you do this.
>
> You can interact with a Buffer object like any other Python object
> implementing the buffer protocol. See numpy.frombuffer as an example
> of a function that interacts with such objects. I would suggest adding
> a method to xnd method for this.
>
> If you need to interact with Plasma from C then things will be more
> complicated -- Robert or Philipp should be able to advise in this
> case.
>
> - Wes
>
> On Tue, Mar 6, 2018 at 4:55 PM, Saul Shanabrook 
> wrote:
> > Hey Wes,
> >
> > I don't have much experience doing C + Python + Cython development, so I
> am
> > probably missing something obvious, but reading the Cython docs,
> > <
> https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#public-declarations
> >
> > it
> > seems like I can only access types marked as public from C code. When I
> > compile arrow locally, I do get some C++ headers for the plasma code,
> but I
> > don't think I can use them from C code either.
> >
> > Best,
> > Saul
> >
> >
> >
> > On Tue, Mar 6, 2018 at 3:12 PM Wes McKinney  wrote:
> >
> >> hi Saul,
> >>
> >> Are you able to use the buffer/memoryview protocol? Instances of
> >> pyarrow.Buffer, like PlasmaBuffer, support this
> >>
> >>
> https://github.com/apache/arrow/blob/master/python/pyarrow/plasma.pyx#L182
> >>
> >> - Wes
> >>
> >> On Tue, Mar 6, 2018 at 3:09 PM, Saul Shanabrook  >
> >> wrote:
> >> > I am trying to use the Plasma store to back xnd objects. Xnd (
> >> > https://xnd.readthedocs.io/en/latest/xnd/index.html) is a container
> >> library
> >> > in C that has Python bindings. I would like to get a pointer to the
> >> > allocated memory after creating or get an object in Plasma. I see that
> >> this
> >> > is supported in the C++ API (
> >> >
> >>
> https://arrow.apache.org/docs/cpp/classplasma_1_1_plasma_client.html#ac18ab9cc792c620a97a3dcb165e0ecd7
> >> )
> >> > but not in the python API (as far as I can tell). Is it possible to
> use
> >> the
> >> > C++ Plasma API from a C project? If not, would it make sense to expose
> >> > pointer access on the Python API using capsules
> >> > https://docs.python.org/3.6/c-api/capsule.html
> >> > ?
> >>
>

Re: Using Plasma with xnd

[Wes McKinney]

hi Saul -- I think the easiest solution here is the buffer/memoryview
protocol. You won't have to touch the Cython or C++ API from pyarrow
if you do this.

You can interact with a Buffer object like any other Python object
implementing the buffer protocol. See numpy.frombuffer as an example
of a function that interacts with such objects. I would suggest adding
a method to xnd method for this.

If you need to interact with Plasma from C then things will be more
complicated -- Robert or Philipp should be able to advise in this
case.

- Wes

On Tue, Mar 6, 2018 at 4:55 PM, Saul Shanabrook  wrote:
> Hey Wes,
>
> I don't have much experience doing C + Python + Cython development, so I am
> probably missing something obvious, but reading the Cython docs,
> 
> it
> seems like I can only access types marked as public from C code. When I
> compile arrow locally, I do get some C++ headers for the plasma code, but I
> don't think I can use them from C code either.
>
> Best,
> Saul
>
>
>
> On Tue, Mar 6, 2018 at 3:12 PM Wes McKinney  wrote:
>
>> hi Saul,
>>
>> Are you able to use the buffer/memoryview protocol? Instances of
>> pyarrow.Buffer, like PlasmaBuffer, support this
>>
>> https://github.com/apache/arrow/blob/master/python/pyarrow/plasma.pyx#L182
>>
>> - Wes
>>
>> On Tue, Mar 6, 2018 at 3:09 PM, Saul Shanabrook 
>> wrote:
>> > I am trying to use the Plasma store to back xnd objects. Xnd (
>> > https://xnd.readthedocs.io/en/latest/xnd/index.html) is a container
>> library
>> > in C that has Python bindings. I would like to get a pointer to the
>> > allocated memory after creating or get an object in Plasma. I see that
>> this
>> > is supported in the C++ API (
>> >
>> https://arrow.apache.org/docs/cpp/classplasma_1_1_plasma_client.html#ac18ab9cc792c620a97a3dcb165e0ecd7
>> )
>> > but not in the python API (as far as I can tell). Is it possible to use
>> the
>> > C++ Plasma API from a C project? If not, would it make sense to expose
>> > pointer access on the Python API using capsules
>> > https://docs.python.org/3.6/c-api/capsule.html
>> > ?
>>

Re: Using Plasma with xnd

[Saul Shanabrook]

Hey Wes,

I don't have much experience doing C + Python + Cython development, so I am
probably missing something obvious, but reading the Cython docs,

it
seems like I can only access types marked as public from C code. When I
compile arrow locally, I do get some C++ headers for the plasma code, but I
don't think I can use them from C code either.

Best,
Saul



On Tue, Mar 6, 2018 at 3:12 PM Wes McKinney  wrote:

> hi Saul,
>
> Are you able to use the buffer/memoryview protocol? Instances of
> pyarrow.Buffer, like PlasmaBuffer, support this
>
> https://github.com/apache/arrow/blob/master/python/pyarrow/plasma.pyx#L182
>
> - Wes
>
> On Tue, Mar 6, 2018 at 3:09 PM, Saul Shanabrook 
> wrote:
> > I am trying to use the Plasma store to back xnd objects. Xnd (
> > https://xnd.readthedocs.io/en/latest/xnd/index.html) is a container
> library
> > in C that has Python bindings. I would like to get a pointer to the
> > allocated memory after creating or get an object in Plasma. I see that
> this
> > is supported in the C++ API (
> >
> https://arrow.apache.org/docs/cpp/classplasma_1_1_plasma_client.html#ac18ab9cc792c620a97a3dcb165e0ecd7
> )
> > but not in the python API (as far as I can tell). Is it possible to use
> the
> > C++ Plasma API from a C project? If not, would it make sense to expose
> > pointer access on the Python API using capsules
> > https://docs.python.org/3.6/c-api/capsule.html
> > ?
>

Re: Using Plasma with xnd

[Wes McKinney]

hi Saul,

Are you able to use the buffer/memoryview protocol? Instances of
pyarrow.Buffer, like PlasmaBuffer, support this

https://github.com/apache/arrow/blob/master/python/pyarrow/plasma.pyx#L182

- Wes

On Tue, Mar 6, 2018 at 3:09 PM, Saul Shanabrook  wrote:
> I am trying to use the Plasma store to back xnd objects. Xnd (
> https://xnd.readthedocs.io/en/latest/xnd/index.html) is a container library
> in C that has Python bindings. I would like to get a pointer to the
> allocated memory after creating or get an object in Plasma. I see that this
> is supported in the C++ API (
> https://arrow.apache.org/docs/cpp/classplasma_1_1_plasma_client.html#ac18ab9cc792c620a97a3dcb165e0ecd7)
> but not in the python API (as far as I can tell). Is it possible to use the
> C++ Plasma API from a C project? If not, would it make sense to expose
> pointer access on the Python API using capsules
> https://docs.python.org/3.6/c-api/capsule.html
> ?

Using Plasma with xnd

[Saul Shanabrook]

I am trying to use the Plasma store to back xnd objects. Xnd (
https://xnd.readthedocs.io/en/latest/xnd/index.html) is a container library
in C that has Python bindings. I would like to get a pointer to the
allocated memory after creating or get an object in Plasma. I see that this
is supported in the C++ API (
https://arrow.apache.org/docs/cpp/classplasma_1_1_plasma_client.html#ac18ab9cc792c620a97a3dcb165e0ecd7)
but not in the python API (as far as I can tell). Is it possible to use the
C++ Plasma API from a C project? If not, would it make sense to expose
pointer access on the Python API using capsules
https://docs.python.org/3.6/c-api/capsule.html
?