Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-20 Thread Victor Stinner
* Add new GIL-free (no need to hold the GIL) memory allocator functions: - ``void* PyMem_RawMalloc(size_t size)`` - ``void* PyMem_RawRealloc(void *ptr, size_t new_size)`` - ``void PyMem_RawFree(void *ptr)`` - the behaviour of requesting zero bytes is not defined: return *NULL*

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-20 Thread Kristján Valur Jónsson
:59 To: Kristján Valur Jónsson Cc: Python Dev Subject: Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators Is PyMemMappingAllocator complete enough for your usage at CCP Games? ___ Python-Dev mailing list Python-Dev

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-20 Thread Nick Coghlan
On 20 June 2013 15:37, Victor Stinner victor.stin...@gmail.com wrote: Le jeudi 20 juin 2013, Nick Coghlan a écrit : Is PyMemMappingAllocator complete enough for your usage at CCP Games? Can we go back to calling this the Arena allocator? Or at least Mapped? When I see Mapping in the context

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Antoine Pitrou
Le Tue, 18 Jun 2013 22:40:49 +0200, Victor Stinner victor.stin...@gmail.com a écrit : Other changes - [...] * Configure external libraries like zlib or OpenSSL to allocate memory using ``PyMem_RawMalloc()`` Why so, and is it done by default? Only one get/set function for

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Victor Stinner
2013/6/19 Antoine Pitrou solip...@pitrou.net: Le Tue, 18 Jun 2013 22:40:49 +0200, Victor Stinner victor.stin...@gmail.com a écrit : Other changes - [...] * Configure external libraries like zlib or OpenSSL to allocate memory using ``PyMem_RawMalloc()`` Why so, and is it

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Kristján Valur Jónsson
] On Behalf Of Scott Dial Sent: 19. júní 2013 04:34 To: ncogh...@gmail.com Cc: Python-Dev@python.org Subject: Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators On 6/18/2013 11:32 PM, Nick Coghlan wrote: Agreed more of that rationale needs to be moved from

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Antoine Pitrou
On Wed, 19 Jun 2013 17:24:21 +0200 Victor Stinner victor.stin...@gmail.com wrote: For the track memory usage use case, it is important to track memory allocated in external libraries to have accurate reports, because these allocations may be huge. [...] Not in main(). The Python expat and

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Victor Stinner
2013/6/19 Antoine Pitrou solip...@pitrou.net: On Wed, 19 Jun 2013 17:24:21 +0200 Drawback: the caller has to check if the result is 0, or handle the error. Or you can just call Py_FatalError() if the domain is invalid. I don't like Py_FatalError(), especially when Python is embedded.

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Kristján Valur Jónsson
: PEP 445: Add new APIs to customize Python memory allocators typedef struct { /* user context passed as the first argument to the 2 functions */ void *ctx; /* allocate a memory mapping */ void* (*alloc) (void *ctx, size_t size

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Victor Stinner
2013/6/19 Kristján Valur Jónsson krist...@ccpgames.com: Oh, it should be public, in my opinion. Ok. And do you think that the PyMemMappingAllocator structure is complete, or that we should add something to be future-proof? At least, PyMemMappingAllocator is enough for pymalloc usage :-) Is

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Terry Reedy
On 6/19/2013 11:24 AM, Victor Stinner wrote: 2013/6/19 Antoine Pitrou solip...@pitrou.net: Le Tue, 18 Jun 2013 22:40:49 +0200, Victor Stinner victor.stin...@gmail.com a écrit : Only one get/set function for block allocators -- Replace the 6

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Antoine Pitrou
On Wed, 19 Jun 2013 17:49:02 +0200 Victor Stinner victor.stin...@gmail.com wrote: 2013/6/19 Antoine Pitrou solip...@pitrou.net: On Wed, 19 Jun 2013 17:24:21 +0200 Drawback: the caller has to check if the result is 0, or handle the error. Or you can just call Py_FatalError() if the

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Victor Stinner
PyMem_RawAlloc()/Realloc/Free should be part of the stable ABI. I agree that all other new fumctions ans structures should not. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Nick Coghlan
On 20 Jun 2013 02:03, Victor Stinner victor.stin...@gmail.com wrote: 2013/6/19 Kristján Valur Jónsson krist...@ccpgames.com: Oh, it should be public, in my opinion. Ok. And do you think that the PyMemMappingAllocator structure is complete, or that we should add something to be future-proof?

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Victor Stinner
Le jeudi 20 juin 2013, Nick Coghlan a écrit : Is PyMemMappingAllocator complete enough for your usage at CCP Games? Can we go back to calling this the Arena allocator? Or at least Mapped? When I see Mapping in the context of Python I think of the container API, not a memory allocation API.

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-18 Thread Victor Stinner
typedef struct { /* user context passed as the first argument to the 2 functions */ void *ctx; /* allocate a memory mapping */ void* (*alloc) (void *ctx, size_t size); /* release a memory mapping */ void (*free) (void *ctx, void

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-18 Thread Scott Dial
On 6/18/2013 4:40 PM, Victor Stinner wrote: No context argument --- Simplify the signature of allocator functions, remove the context argument: * ``void* malloc(size_t size)`` * ``void* realloc(void *ptr, size_t new_size)`` * ``void free(void *ptr)`` It is likely for

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-18 Thread Nick Coghlan
On 19 June 2013 09:23, Scott Dial scott+python-...@scottdial.com wrote: On 6/18/2013 4:40 PM, Victor Stinner wrote: No context argument --- Simplify the signature of allocator functions, remove the context argument: * ``void* malloc(size_t size)`` * ``void* realloc(void

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-18 Thread Scott Dial
On 6/18/2013 11:32 PM, Nick Coghlan wrote: Agreed more of that rationale needs to be moved from the issue tracker into the PEP, though. Thanks for the clarification. I hadn't read the issue tracker at all. On it's face value, I didn't see what purpose it served, but having read Kristján's

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-18 Thread Victor Stinner
Le mercredi 19 juin 2013, Scott Dial a écrit : On 6/18/2013 4:40 PM, Victor Stinner wrote: No context argument I think there is a lack of justification for the extra argument, and the extra argument is not free. The typical use-case for doing this continuation-passing style is when the set