Re: [julia-users] What is the proper way to allocate and free C struct?

[Andrei Zh]

It makes sense. Thanks, once again!

For reference, I ended up with code like this:

# internal only
type XMatrixStruct
  rows::Cint
  cols::Cint
  data::Ptr{Float64}
end


type XMatrix
  ptr::Ptr{XMatrixStruct}
  data::Ptr{Float64}   # copied from `unsafe_load(ptr).data`
end

This way I can manipulate original pointer and also have direct access to 
the `data` field. 



On Sunday, April 17, 2016 at 3:35:34 PM UTC+3, Yichao Yu wrote:
>
> On Sun, Apr 17, 2016 at 6:23 AM, Andrei Zh  > wrote: 
> > A followup question. In the above example I'd like to add method 
> `getindex` 
> > for `XMatrix`. Previously, I could do it like this: 
> > 
> > type XMatrix 
> >   rows::Cint 
> >   cols::Cint 
> >   data::Ptr{Float64} 
> > end 
> > 
> > getindex(mat::XMatrix, i::Integer) = unsafe_load(mat.data, i) 
> > 
> > Now after Yichao's suggestion my type looks like: 
> > 
> > type XMatrix 
> >   ptr::Ptr{Void} 
> > end 
> > 
> > So I don't have direct access to the `data` field.  What is the proper 
> way 
> > to get it? 
> > 
> > So far my best idea is to use something like this: 
> > 
> > type XMatrixStruct 
> >   rows::Cint 
> >   cols::Cint 
> >   data::Ptr{Float64} 
> > end 
>
> ^^ I would make this an `immutable` 
>
> > 
> > type XMatrix 
> >   ptr::Ptr{XMatrixStruct} 
> > end 
> > 
> > function getindex(mat::XMatrix, i::Integer) 
> >   struct = unsafe_load(mat.ptr) # load copy of XMatrixStruct 
> > type 
>
> And then llvm should optimize out the load of the entire struct. 
>
> >   value = unsafe_load(struct.data, i) # load value from its data 
> field 
> > end 
>
> There are of course still cases that is inconvenient (e.g. if you want 
> to store to a field) something like this[1] may help with that. In the 
> mean time, you can load and store the field directly with an offset 
> too. 
>
> [1] https://github.com/JuliaLang/julia/issues/11902 
>
> > 
> > But it would be nice to avoid intermediate `XMatrixStruct` type. 
> > 
> > 
> > 
> > On Sunday, April 17, 2016 at 3:49:51 AM UTC+3, Andrei Zh wrote: 
> >> 
> >> This is exactly the answer I hoped to see! Thanks a lot! 
> >> 
> >> On Sunday, April 17, 2016 at 2:46:15 AM UTC+3, Yichao Yu wrote: 
> >>> 
> >>> On Sat, Apr 16, 2016 at 6:55 PM, Andrei Zh  
> wrote: 
> >>> > I have a question regarding Struct Type correspondences section of 
> the 
> >>> > documentation. Let's say, in a C library we have a struct like this: 
> >>> > 
> >>> > typedef struct { 
> >>> >   int rows; 
> >>> >   int cols; 
> >>> >   double* data; 
> >>> > } xmatrix; 
> >>> > 
> >>> > In C code it can be created using function: 
> >>> > 
> >>> > int xmatrix_copy(void* src, xmatrix** mat, int rows, int cols); 
> >>> > 
> >>> > Note pointer-to-pointer in second argument. This C function 
> allocates 
> >>> > new 
> >>> > region of memory for `xmatrix` and fills in the values (rows, cols 
> and 
> >>> > pointer to data). I call this function from Julia like this: 
> >>> > 
> >>> > type XMatrix 
> >>> >   rows::Cint 
> >>> >   cols::Cint 
> >>> >   data::Ptr{Float64} 
> >>> > end 
> >>> > 
> >>> > 
> >>> > data = Float64[1, 2, 3, 4, 5, 6] 
> >>> > matptr = Array(Ptr{XMatrix}, 1) 
> >>> > 
> >>> > 
> >>> > ccall((:xmatrix_copy), Cint, 
> >>> > (Ptr{Void}, Ptr{Ptr{XMatrix}}, Cint, Cint), 
> >>> > pointer(data), matptr, Cint(3), Cint(2)) 
> >>> > 
> >>> > mat = unsafe_load(matptr[1]) 
> >>> 
> >>> This is wrong, it leaks memory. See below. 
> >>> 
> >>> > 
> >>> > Details to notice in `ccall`: 
> >>> > 
> >>> > 1. In type tuple I set type of xmatrix argument to be a 
> >>> > pointer-to-pointer 
> >>> > to XMatrix. 
> >>> > 2. As an argument, I pass 1-element array of pointers to XMatrix and 
> >>> > let C 
> >>> > code to fill it in. 
> >>> > 
> >>> > This approach works and I get initialized `XMatrix`. Yet, I don't 
> >>> > really 
> >>> > understand all details of memory management in this case, In 
> >>> > particular: 
> >>> > 
> >>> > 1. Who owns memory of an instance of `XMatrix`? Is `unsafe_load` 
> >>> > copying 
> >>> > fields or just assigns Julia type tag to C-allocated region of 
> memory? 
> >>> 
> >>> The owner of the memory is determined by the c library. In this case 
> >>> it seems that the C library owns the memory (which is generally the 
> >>> case if the library provide new and free functions) 
> >>> Julia never (and can never) assign tag to C memory. Unsafe load does 
> >>> exactly what the name suggests and is equivalent to `*ptr` in C. It 
> >>> copies the content of the struct and the return value 
> >>> has no aliasing with the original pointer. 
> >>> 
> >>> I assume the C library want the original pointer so you need to keep 
> >>> the pointer in the wrapper type and define conversion functions to 
> >>> pass it to C. See this[1] for an example for the proper way to do it 
> >>> (mostly the ptr field, the cconvert and unsafe_convert definitions and 
> >>> the finalizer(explained below)). 
> 

Re: [julia-users] What is the proper way to allocate and free C struct?

[Yichao Yu]

On Sun, Apr 17, 2016 at 6:23 AM, Andrei Zh  wrote:
> A followup question. In the above example I'd like to add method `getindex`
> for `XMatrix`. Previously, I could do it like this:
>
> type XMatrix
>   rows::Cint
>   cols::Cint
>   data::Ptr{Float64}
> end
>
> getindex(mat::XMatrix, i::Integer) = unsafe_load(mat.data, i)
>
> Now after Yichao's suggestion my type looks like:
>
> type XMatrix
>   ptr::Ptr{Void}
> end
>
> So I don't have direct access to the `data` field.  What is the proper way
> to get it?
>
> So far my best idea is to use something like this:
>
> type XMatrixStruct
>   rows::Cint
>   cols::Cint
>   data::Ptr{Float64}
> end

^^ I would make this an `immutable`

>
> type XMatrix
>   ptr::Ptr{XMatrixStruct}
> end
>
> function getindex(mat::XMatrix, i::Integer)
>   struct = unsafe_load(mat.ptr) # load copy of XMatrixStruct
> type

And then llvm should optimize out the load of the entire struct.

>   value = unsafe_load(struct.data, i) # load value from its data field
> end

There are of course still cases that is inconvenient (e.g. if you want
to store to a field) something like this[1] may help with that. In the
mean time, you can load and store the field directly with an offset
too.

[1] https://github.com/JuliaLang/julia/issues/11902

>
> But it would be nice to avoid intermediate `XMatrixStruct` type.
>
>
>
> On Sunday, April 17, 2016 at 3:49:51 AM UTC+3, Andrei Zh wrote:
>>
>> This is exactly the answer I hoped to see! Thanks a lot!
>>
>> On Sunday, April 17, 2016 at 2:46:15 AM UTC+3, Yichao Yu wrote:
>>>
>>> On Sat, Apr 16, 2016 at 6:55 PM, Andrei Zh  wrote:
>>> > I have a question regarding Struct Type correspondences section of the
>>> > documentation. Let's say, in a C library we have a struct like this:
>>> >
>>> > typedef struct {
>>> >   int rows;
>>> >   int cols;
>>> >   double* data;
>>> > } xmatrix;
>>> >
>>> > In C code it can be created using function:
>>> >
>>> > int xmatrix_copy(void* src, xmatrix** mat, int rows, int cols);
>>> >
>>> > Note pointer-to-pointer in second argument. This C function allocates
>>> > new
>>> > region of memory for `xmatrix` and fills in the values (rows, cols and
>>> > pointer to data). I call this function from Julia like this:
>>> >
>>> > type XMatrix
>>> >   rows::Cint
>>> >   cols::Cint
>>> >   data::Ptr{Float64}
>>> > end
>>> >
>>> >
>>> > data = Float64[1, 2, 3, 4, 5, 6]
>>> > matptr = Array(Ptr{XMatrix}, 1)
>>> >
>>> >
>>> > ccall((:xmatrix_copy), Cint,
>>> > (Ptr{Void}, Ptr{Ptr{XMatrix}}, Cint, Cint),
>>> > pointer(data), matptr, Cint(3), Cint(2))
>>> >
>>> > mat = unsafe_load(matptr[1])
>>>
>>> This is wrong, it leaks memory. See below.
>>>
>>> >
>>> > Details to notice in `ccall`:
>>> >
>>> > 1. In type tuple I set type of xmatrix argument to be a
>>> > pointer-to-pointer
>>> > to XMatrix.
>>> > 2. As an argument, I pass 1-element array of pointers to XMatrix and
>>> > let C
>>> > code to fill it in.
>>> >
>>> > This approach works and I get initialized `XMatrix`. Yet, I don't
>>> > really
>>> > understand all details of memory management in this case, In
>>> > particular:
>>> >
>>> > 1. Who owns memory of an instance of `XMatrix`? Is `unsafe_load`
>>> > copying
>>> > fields or just assigns Julia type tag to C-allocated region of memory?
>>>
>>> The owner of the memory is determined by the c library. In this case
>>> it seems that the C library owns the memory (which is generally the
>>> case if the library provide new and free functions)
>>> Julia never (and can never) assign tag to C memory. Unsafe load does
>>> exactly what the name suggests and is equivalent to `*ptr` in C. It
>>> copies the content of the struct and the return value
>>> has no aliasing with the original pointer.
>>>
>>> I assume the C library want the original pointer so you need to keep
>>> the pointer in the wrapper type and define conversion functions to
>>> pass it to C. See this[1] for an example for the proper way to do it
>>> (mostly the ptr field, the cconvert and unsafe_convert definitions and
>>> the finalizer(explained below)).
>>>
>>> > 2. What would be the proper way to free allocated memory given that C
>>> > library provides function `xmatrix_free(xmatrix* mat)`?
>>>
>>> ccall the free function. You usually also want a finalizer on the
>>> wrapper type mentioned above so that the GC can free the memory for
>>> you when the julia object is collected. Note that you should use
>>> finalizer to finalize object only for types that you don't care too
>>> much about their lifetime since the time when a finalizer is called is
>>> undefined.
>>>
>>> > 3. Aforementioned section (and the next one) suggest to use
>>> > `Ref{XMatrix}`
>>> > instead. Can somebody provide an example of corresponding code using
>>> > refs
>>> > instead of wrapping into an array?
>>>
>>> Ref on a bits type is basically a 0-dim array
>>>
>>> matout = Ref{Ptr{XMatrix}}()
>>>
>>> ccall((:xmatrix_copy), 

Re: [julia-users] What is the proper way to allocate and free C struct?

[Andrei Zh]

A followup question. In the above example I'd like to add method `getindex` 
for `XMatrix`. Previously, I could do it like this:

type XMatrix
  rows::Cint
  cols::Cint
  data::Ptr{Float64}
end

getindex(mat::XMatrix, i::Integer) = unsafe_load(mat.data, i)

Now after Yichao's suggestion my type looks like:

type XMatrix
  ptr::Ptr{Void}
end

So I don't have direct access to the `data` field.  What is the proper way 
to get it? 

So far my best idea is to use something like this: 

type XMatrixStruct
  rows::Cint
  cols::Cint
  data::Ptr{Float64}
end

type XMatrix
  ptr::Ptr{XMatrixStruct}
end

function getindex(mat::XMatrix, i::Integer)
  struct = unsafe_load(mat.ptr) # load copy of XMatrixStruct 
type
  value = unsafe_load(struct.data, i) # load value from its data field
end

But it would be nice to avoid intermediate `XMatrixStruct` type.



On Sunday, April 17, 2016 at 3:49:51 AM UTC+3, Andrei Zh wrote:
>
> This is exactly the answer I hoped to see! Thanks a lot!
>
> On Sunday, April 17, 2016 at 2:46:15 AM UTC+3, Yichao Yu wrote:
>>
>> On Sat, Apr 16, 2016 at 6:55 PM, Andrei Zh  wrote: 
>> > I have a question regarding Struct Type correspondences section of the 
>> > documentation. Let's say, in a C library we have a struct like this: 
>> > 
>> > typedef struct { 
>> >   int rows; 
>> >   int cols; 
>> >   double* data; 
>> > } xmatrix; 
>> > 
>> > In C code it can be created using function: 
>> > 
>> > int xmatrix_copy(void* src, xmatrix** mat, int rows, int cols); 
>> > 
>> > Note pointer-to-pointer in second argument. This C function allocates 
>> new 
>> > region of memory for `xmatrix` and fills in the values (rows, cols and 
>> > pointer to data). I call this function from Julia like this: 
>> > 
>> > type XMatrix 
>> >   rows::Cint 
>> >   cols::Cint 
>> >   data::Ptr{Float64} 
>> > end 
>> > 
>> > 
>> > data = Float64[1, 2, 3, 4, 5, 6] 
>> > matptr = Array(Ptr{XMatrix}, 1) 
>> > 
>> > 
>> > ccall((:xmatrix_copy), Cint, 
>> > (Ptr{Void}, Ptr{Ptr{XMatrix}}, Cint, Cint), 
>> > pointer(data), matptr, Cint(3), Cint(2)) 
>> > 
>> > mat = unsafe_load(matptr[1]) 
>>
>> This is wrong, it leaks memory. See below. 
>>
>> > 
>> > Details to notice in `ccall`: 
>> > 
>> > 1. In type tuple I set type of xmatrix argument to be a 
>> pointer-to-pointer 
>> > to XMatrix. 
>> > 2. As an argument, I pass 1-element array of pointers to XMatrix and 
>> let C 
>> > code to fill it in. 
>> > 
>> > This approach works and I get initialized `XMatrix`. Yet, I don't 
>> really 
>> > understand all details of memory management in this case, In 
>> particular: 
>> > 
>> > 1. Who owns memory of an instance of `XMatrix`? Is `unsafe_load` 
>> copying 
>> > fields or just assigns Julia type tag to C-allocated region of memory? 
>>
>> The owner of the memory is determined by the c library. In this case 
>> it seems that the C library owns the memory (which is generally the 
>> case if the library provide new and free functions) 
>> Julia never (and can never) assign tag to C memory. Unsafe load does 
>> exactly what the name suggests and is equivalent to `*ptr` in C. It 
>> copies the content of the struct and the return value 
>> has no aliasing with the original pointer. 
>>
>> I assume the C library want the original pointer so you need to keep 
>> the pointer in the wrapper type and define conversion functions to 
>> pass it to C. See this[1] for an example for the proper way to do it 
>> (mostly the ptr field, the cconvert and unsafe_convert definitions and 
>> the finalizer(explained below)). 
>>
>> > 2. What would be the proper way to free allocated memory given that C 
>> > library provides function `xmatrix_free(xmatrix* mat)`? 
>>
>> ccall the free function. You usually also want a finalizer on the 
>> wrapper type mentioned above so that the GC can free the memory for 
>> you when the julia object is collected. Note that you should use 
>> finalizer to finalize object only for types that you don't care too 
>> much about their lifetime since the time when a finalizer is called is 
>> undefined. 
>>
>> > 3. Aforementioned section (and the next one) suggest to use 
>> `Ref{XMatrix}` 
>> > instead. Can somebody provide an example of corresponding code using 
>> refs 
>> > instead of wrapping into an array? 
>>
>> Ref on a bits type is basically a 0-dim array 
>>
>> matout = Ref{Ptr{XMatrix}}() 
>>
>> ccall((:xmatrix_copy), Cint, 
>> (Ptr{Void}, Ptr{Ptr{XMatrix}}, Cint, Cint), 
>> data, matout, Cint(3), Cint(2)) # DO NOT USE 
>> `pointer(data)` This is the single most common mistake when using 
>> ccall. It is fine in the global scope but if this is in a function, 
>> julia is free to collect the data array when you are in/before 
>> entering the ccall. See the manual section about cconvert and 
>> unsafe_convert 
>>
>> matptr = matout[] # This is the C managed/owned pointer that you 
>> should embed in your julia type. 
>>
>> > 
>> > Thanks, 
>> > 

Re: [julia-users] What is the proper way to allocate and free C struct?

[Andrei Zh]

This is exactly the answer I hoped to see! Thanks a lot!

On Sunday, April 17, 2016 at 2:46:15 AM UTC+3, Yichao Yu wrote:
>
> On Sat, Apr 16, 2016 at 6:55 PM, Andrei Zh  > wrote: 
> > I have a question regarding Struct Type correspondences section of the 
> > documentation. Let's say, in a C library we have a struct like this: 
> > 
> > typedef struct { 
> >   int rows; 
> >   int cols; 
> >   double* data; 
> > } xmatrix; 
> > 
> > In C code it can be created using function: 
> > 
> > int xmatrix_copy(void* src, xmatrix** mat, int rows, int cols); 
> > 
> > Note pointer-to-pointer in second argument. This C function allocates 
> new 
> > region of memory for `xmatrix` and fills in the values (rows, cols and 
> > pointer to data). I call this function from Julia like this: 
> > 
> > type XMatrix 
> >   rows::Cint 
> >   cols::Cint 
> >   data::Ptr{Float64} 
> > end 
> > 
> > 
> > data = Float64[1, 2, 3, 4, 5, 6] 
> > matptr = Array(Ptr{XMatrix}, 1) 
> > 
> > 
> > ccall((:xmatrix_copy), Cint, 
> > (Ptr{Void}, Ptr{Ptr{XMatrix}}, Cint, Cint), 
> > pointer(data), matptr, Cint(3), Cint(2)) 
> > 
> > mat = unsafe_load(matptr[1]) 
>
> This is wrong, it leaks memory. See below. 
>
> > 
> > Details to notice in `ccall`: 
> > 
> > 1. In type tuple I set type of xmatrix argument to be a 
> pointer-to-pointer 
> > to XMatrix. 
> > 2. As an argument, I pass 1-element array of pointers to XMatrix and let 
> C 
> > code to fill it in. 
> > 
> > This approach works and I get initialized `XMatrix`. Yet, I don't really 
> > understand all details of memory management in this case, In particular: 
> > 
> > 1. Who owns memory of an instance of `XMatrix`? Is `unsafe_load` copying 
> > fields or just assigns Julia type tag to C-allocated region of memory? 
>
> The owner of the memory is determined by the c library. In this case 
> it seems that the C library owns the memory (which is generally the 
> case if the library provide new and free functions) 
> Julia never (and can never) assign tag to C memory. Unsafe load does 
> exactly what the name suggests and is equivalent to `*ptr` in C. It 
> copies the content of the struct and the return value 
> has no aliasing with the original pointer. 
>
> I assume the C library want the original pointer so you need to keep 
> the pointer in the wrapper type and define conversion functions to 
> pass it to C. See this[1] for an example for the proper way to do it 
> (mostly the ptr field, the cconvert and unsafe_convert definitions and 
> the finalizer(explained below)). 
>
> > 2. What would be the proper way to free allocated memory given that C 
> > library provides function `xmatrix_free(xmatrix* mat)`? 
>
> ccall the free function. You usually also want a finalizer on the 
> wrapper type mentioned above so that the GC can free the memory for 
> you when the julia object is collected. Note that you should use 
> finalizer to finalize object only for types that you don't care too 
> much about their lifetime since the time when a finalizer is called is 
> undefined. 
>
> > 3. Aforementioned section (and the next one) suggest to use 
> `Ref{XMatrix}` 
> > instead. Can somebody provide an example of corresponding code using 
> refs 
> > instead of wrapping into an array? 
>
> Ref on a bits type is basically a 0-dim array 
>
> matout = Ref{Ptr{XMatrix}}() 
>
> ccall((:xmatrix_copy), Cint, 
> (Ptr{Void}, Ptr{Ptr{XMatrix}}, Cint, Cint), 
> data, matout, Cint(3), Cint(2)) # DO NOT USE 
> `pointer(data)` This is the single most common mistake when using 
> ccall. It is fine in the global scope but if this is in a function, 
> julia is free to collect the data array when you are in/before 
> entering the ccall. See the manual section about cconvert and 
> unsafe_convert 
>
> matptr = matout[] # This is the C managed/owned pointer that you 
> should embed in your julia type. 
>
> > 
> > Thanks, 
> > Andrei 
>
>
> [1] 
> https://github.com/yuyichao/LibArchive.jl/blob/4112772e8d1d7124c896bf98c5baee13149d3f8d/src/writer.jl#L28-L58
>  
>

Re: [julia-users] What is the proper way to allocate and free C struct?

[Yichao Yu]

On Sat, Apr 16, 2016 at 6:55 PM, Andrei Zh  wrote:
> I have a question regarding Struct Type correspondences section of the
> documentation. Let's say, in a C library we have a struct like this:
>
> typedef struct {
>   int rows;
>   int cols;
>   double* data;
> } xmatrix;
>
> In C code it can be created using function:
>
> int xmatrix_copy(void* src, xmatrix** mat, int rows, int cols);
>
> Note pointer-to-pointer in second argument. This C function allocates new
> region of memory for `xmatrix` and fills in the values (rows, cols and
> pointer to data). I call this function from Julia like this:
>
> type XMatrix
>   rows::Cint
>   cols::Cint
>   data::Ptr{Float64}
> end
>
>
> data = Float64[1, 2, 3, 4, 5, 6]
> matptr = Array(Ptr{XMatrix}, 1)
>
>
> ccall((:xmatrix_copy), Cint,
> (Ptr{Void}, Ptr{Ptr{XMatrix}}, Cint, Cint),
> pointer(data), matptr, Cint(3), Cint(2))
>
> mat = unsafe_load(matptr[1])

This is wrong, it leaks memory. See below.

>
> Details to notice in `ccall`:
>
> 1. In type tuple I set type of xmatrix argument to be a pointer-to-pointer
> to XMatrix.
> 2. As an argument, I pass 1-element array of pointers to XMatrix and let C
> code to fill it in.
>
> This approach works and I get initialized `XMatrix`. Yet, I don't really
> understand all details of memory management in this case, In particular:
>
> 1. Who owns memory of an instance of `XMatrix`? Is `unsafe_load` copying
> fields or just assigns Julia type tag to C-allocated region of memory?

The owner of the memory is determined by the c library. In this case
it seems that the C library owns the memory (which is generally the
case if the library provide new and free functions)
Julia never (and can never) assign tag to C memory. Unsafe load does
exactly what the name suggests and is equivalent to `*ptr` in C. It
copies the content of the struct and the return value
has no aliasing with the original pointer.

I assume the C library want the original pointer so you need to keep
the pointer in the wrapper type and define conversion functions to
pass it to C. See this[1] for an example for the proper way to do it
(mostly the ptr field, the cconvert and unsafe_convert definitions and
the finalizer(explained below)).

> 2. What would be the proper way to free allocated memory given that C
> library provides function `xmatrix_free(xmatrix* mat)`?

ccall the free function. You usually also want a finalizer on the
wrapper type mentioned above so that the GC can free the memory for
you when the julia object is collected. Note that you should use
finalizer to finalize object only for types that you don't care too
much about their lifetime since the time when a finalizer is called is
undefined.

> 3. Aforementioned section (and the next one) suggest to use `Ref{XMatrix}`
> instead. Can somebody provide an example of corresponding code using refs
> instead of wrapping into an array?

Ref on a bits type is basically a 0-dim array

matout = Ref{Ptr{XMatrix}}()

ccall((:xmatrix_copy), Cint,
(Ptr{Void}, Ptr{Ptr{XMatrix}}, Cint, Cint),
data, matout, Cint(3), Cint(2)) # DO NOT USE
`pointer(data)` This is the single most common mistake when using
ccall. It is fine in the global scope but if this is in a function,
julia is free to collect the data array when you are in/before
entering the ccall. See the manual section about cconvert and
unsafe_convert

matptr = matout[] # This is the C managed/owned pointer that you
should embed in your julia type.

>
> Thanks,
> Andrei


[1] 
https://github.com/yuyichao/LibArchive.jl/blob/4112772e8d1d7124c896bf98c5baee13149d3f8d/src/writer.jl#L28-L58