Re: ctypes & allocated memory
Hi,
> But the problem is that by specifying the type as ctypes.c_char_p,
> ctypes will hide that pointer from you and return a Python object
> instead. I'm not sure how ctypes is doing it under the hood, but I
> suspect ctypes is doing it's own strdup of the string on conversion, and
> managing that memory, but the original pointer ends up being lost and
> leaking memory.
Yes, the problem with my code I posted is that resource.getrusage report the
Python allocated memory. Once I've changed to
psutil.Process().memory_info().rss I see the memory leak. The code now looks
like:
---
...
strdup.restype = ctypes.c_void_p
free = libc.free
free.argtypes = [ctypes.c_void_p]
...
for _ in range(n):
o = strdup(data)
s = ctypes.string_at(o).decode('utf-8')
free(o)
---
This seems to fix the leak.
Thanks
--
https://mail.python.org/mailman/listinfo/python-list
Re: ctypes & allocated memory
On 6/7/20 2:25 PM, Barry wrote: >> Does ctypes, when using restype, frees allocated memory? >> >> For example, will the memory allocated by "strdup" be freed after the "del" >> statement? If not, how can I free it? > > See https://linux.die.net/man/3/strdup that tells you to use free() to delete > memory allocated by strdup. > > You must remember the result of strdup and free it at an appropriate time. But the problem is that by specifying the type as ctypes.c_char_p, ctypes will hide that pointer from you and return a Python object instead. I'm not sure how ctypes is doing it under the hood, but I suspect ctypes is doing it's own strdup of the string on conversion, and managing that memory, but the original pointer ends up being lost and leaking memory. The stack exchange link I posted suggests that have ctypes give you a void * pointer, and then cast that to give you a python string while still having the original pointer to free. Is this correct? -- https://mail.python.org/mailman/listinfo/python-list
Re: ctypes & allocated memory
> On 7 Jun 2020, at 14:23, Miki Tebeka wrote:
>
> Hi,
>
> Does ctypes, when using restype, frees allocated memory?
>
> For example, will the memory allocated by "strdup" be freed after the "del"
> statement? If not, how can I free it?
See https://linux.die.net/man/3/strdup that tells you to use free() to delete
memory allocated by strdup.
You must remember the result of strdup and free it at an appropriate time.
Barry
>
> ---
> import ctypes
>
> libc = ctypes.cdll.LoadLibrary('libc.so.6')
> strdup = libc.strdup
> strdup.argtypes = [ctypes.c_char_p]
> strdup.restype = ctypes.c_char_p
>
> out = strdup(b'hello').decode('utf-8')
> print(out) # hello
> del out
> ---
>
> Thanks,
> Miki
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: ctypes & allocated memory
On 6/7/20 7:15 AM, Miki Tebeka wrote:
> Hi,
>
> Does ctypes, when using restype, frees allocated memory?
>
> For example, will the memory allocated by "strdup" be freed after the "del"
> statement? If not, how can I free it?
I don't think so. I did a quick google search and came up with this
discussion, which may be relevant:
https://stackoverflow.com/questions/13445568/python-ctypes-how-to-free-memory-getting-invalid-pointer-error
>
> ---
> import ctypes
>
> libc = ctypes.cdll.LoadLibrary('libc.so.6')
> strdup = libc.strdup
> strdup.argtypes = [ctypes.c_char_p]
> strdup.restype = ctypes.c_char_p
>
> out = strdup(b'hello').decode('utf-8')
> print(out) # hello
> del out
> ---
>
> Thanks,
> Miki
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: ctypes & allocated memory
> Does ctypes, when using restype, frees allocated memory?
>
> For example, will the memory allocated by "strdup" be freed after the "del"
> statement? If not, how can I free it?
I've tried the following program and I'm more confused now :) Can anyone
explain the output?
---
import ctypes
import gc
import resource
def mem_usage():
return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
libc = ctypes.cdll.LoadLibrary('libc.so.6')
strdup = libc.strdup
strdup.argtypes = [ctypes.c_char_p]
strdup.restype = ctypes.c_char_p
size = 1 << 20
print(f'size: {size:,}')
data = b'x' * size # 1MB
mb = mem_usage()
print(f'memory before: {mb:,}')
n = 1000
print(f'n: {n:,}')
for _ in range(n):
strdup(data)
gc.collect()
ma = mem_usage()
diff = ma - mb
print(f'memory after: {ma:,}')
print(f'diff: {diff:,}')
print(f'diff/size: {diff/size:.2f}')
---
Which prints
---
size: 1,048,576
memory before: 21,556
n: 1,000
memory after: 1,035,180
diff: 1,013,624
diff/size: 0.97
---
--
https://mail.python.org/mailman/listinfo/python-list
ctypes & allocated memory
Hi,
Does ctypes, when using restype, frees allocated memory?
For example, will the memory allocated by "strdup" be freed after the "del"
statement? If not, how can I free it?
---
import ctypes
libc = ctypes.cdll.LoadLibrary('libc.so.6')
strdup = libc.strdup
strdup.argtypes = [ctypes.c_char_p]
strdup.restype = ctypes.c_char_p
out = strdup(b'hello').decode('utf-8')
print(out) # hello
del out
---
Thanks,
Miki
--
https://mail.python.org/mailman/listinfo/python-list
