My feeling is that we should advise users of the library that any
slices of a ResizableBuffer become invalid after a call to Resize.

> I was thinking about something like this [0]. The point is, that the slice
> user has no way of knowing if the slice can still be safely used and who
> owns the memory.

You can look at the Buffer parent to see if there is a parent-child
relationship, which at least tells you whether you definitely do _not_
own the memory.

I'm not convinced from this use case that we need to change the way
that the Buffer abstraction works. If there is a need for memory
ownership-nannying, that may be best handled by some other kind of
abstract interface that uses Buffers for its implementation.

- Wes

On Wed, Apr 11, 2018 at 8:05 AM, Antoine Pitrou <anto...@python.org> wrote:
>
> Hi Dimitri,
>
> Le 11/04/2018 à 13:42, Dimitri Vorona a écrit :
>>
>> I was thinking about something like this [0]. The point is, that the slice
>> user has no way of knowing if the slice can still be safely used and who
>> owns the memory.
>
> I think the answer is that calling free() on something you exported to
> consumers is incorrect.  If you allocate buffers, you should choose a
> Buffer implementation with proper ownership semantics.  For example, we
> have PoolBuffer, but also Python buffers and CUDA buffers.  They all
> (should) have proper ownership.  If you want to create buffers with data
> managed with malloc/free, you need to write a MallocBuffer implementation.
>
>> A step back is a good idea. My use case would be to return a partially
>> built slice on a buffer, while continuing appending to the buffer. Think
>> delta dictionaries: while a slice of the coding table can be sent, we will
>> have additional data to append later on.
>
> I don't know anything about delta dictionaries, but I get the idea.
>
> Does the implementation become harder if you split the coding table into
> several buffers that never get resized?
>
>> To build on your previous proposal: maybe some more finely grained locking
>> mechanism, like the data_ being a shared_ptr<uint_8*>, slices grabbing a
>> copy of it when they want to use it and releasing it afterwards? The parent
>> would then check the couter of the shared_ptr (similar to the number of
>> slices).
>
> You need an actual lock to avoid race conditions (the parent may find a
> zero shared_ptr counter, but another thread would grab a data pointer
> immediately after).
>
> I wonder if we really want such implementation complexity.  Also,
> everyone is now paying the price of locking.  Ideally slicing and
> fetching a data pointer should be cheap.  I'd like to know what others
> think about this.
>
> Regards
>
> Antoine.

Reply via email to