Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

Hi, My pull request has been merged into numpy. numpy now uses PyMem_RawMalloc() rather than PyMem_Malloc() since it uses the memory allocator without holding the GIL: https://github.com/numpy/numpy/pull/7404 It was proposed to modify numpy to hold the GIL. Maybe it will be done later. It means

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

Ping? Is someone still opposed to my change #26249 "Change PyMem_Malloc to use pymalloc allocator"? If no, I think that I will push my change. My change only changes two lines, so it can be easily reverted before CPython 3.6 if we detect major issues in third-party extensions. And maybe it's

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

So what do you think? Is it worth to change PyMem_Malloc() allocator to pymalloc for a small speedup? Should we do something else before doing that? Or do you expect that too many applications use PyMem_Malloc() without holding the GIL and will not run try to run their application with

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-12 14:31 GMT+01:00 M.-A. Lemburg : >>> If your program has bugs, you can use a debug build of Python 3.5 to >>> detect misusage of the API. > > Yes, but people don't necessarily do this, e.g. I have > for a very long time ignored debug builds completely > and when I

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-03-09 18:54 GMT+01:00 Brett Cannon : >>> https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks >> >> The main advantage of this variable is that you don't have to >> recompile Python in debug mode to benefit of these checks. > > I just wanted to say this all

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

On Wed, 9 Mar 2016 at 06:57 Victor Stinner wrote: > 2016-02-08 15:18 GMT+01:00 Victor Stinner : > >> Perhaps if you add some guards somewhere :-) > > > > We have runtime checks but only implemented in debug mode for efficiency. > > > > By the

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-08 15:18 GMT+01:00 Victor Stinner : >> Perhaps if you add some guards somewhere :-) > > We have runtime checks but only implemented in debug mode for efficiency. > > By the way, I proposed once to add an environment variable to allow to > enable these checks

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

ping? 2016-02-08 15:18 GMT+01:00 Victor Stinner : > 2016-02-04 15:05 GMT+01:00 M.-A. Lemburg : >> Sometimes, yes, but we also do allocations for e.g. >> parsing values in Python argument tuples (e.g. using >> "es" or "et"): >> >>

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

Hi, 2016-02-12 14:31 GMT+01:00 M.-A. Lemburg : > Sorry, your email must gotten lost in my inbox. no problemo > Yes, but those are part of the stdlib. You'd need to check > a few C extensions which are not tested as part of the stdlib, > e.g. numpy, scipy, lxml, pillow, etc.

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

On 12.02.2016 12:18, Victor Stinner wrote: > ping? Sorry, your email must gotten lost in my inbox. > 2016-02-08 15:18 GMT+01:00 Victor Stinner : >> 2016-02-04 15:05 GMT+01:00 M.-A. Lemburg : >>> Sometimes, yes, but we also do allocations for e.g. >>>

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-07 9:22 GMT+01:00 Stefan Behnel : > Note that the PyObject_Malloc() functions have never been documented. Yeah, there is an old bug to track this: http://bugs.python.org/issue20064 > And, for example, the "what's new in 2.5" document says: > > """ > Python’s API has

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

2016-02-04 15:05 GMT+01:00 M.-A. Lemburg : > Sometimes, yes, but we also do allocations for e.g. > parsing values in Python argument tuples (e.g. using > "es" or "et"): > > https://docs.python.org/3.6/c-api/arg.html > > We do document to use PyMem_Free() on those; not sure whether

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

M.-A. Lemburg schrieb am 04.02.2016 um 13:54: > On 04.02.2016 13:29, Victor Stinner wrote: >> But, why not PyObject_Malloc() & PObject_Free() were not used in the >> first place? > > Good question. I guess developers simply thought of PyObject_Malloc() > being for PyObjects, not arbitrary memory

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

On 03.02.2016 22:03, Victor Stinner wrote: > Hi, > > There is an old discussion about the performance of PyMem_Malloc() > memory allocator. CPython is stressing a lot memory allocators. Last > time I made statistics, it was for the PEP 454: > "For example, the Python test suites calls malloc() ,

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

On 04.02.2016 13:29, Victor Stinner wrote: > Hi, > > 2016-02-04 11:17 GMT+01:00 M.-A. Lemburg : >>> Do you see any drawback of using pymalloc for PyMem_Malloc()? >> >> Yes: You cannot free memory allocated using pymalloc with the >> standard C lib free(). > > That's not

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

Hi, 2016-02-04 11:17 GMT+01:00 M.-A. Lemburg : >> Do you see any drawback of using pymalloc for PyMem_Malloc()? > > Yes: You cannot free memory allocated using pymalloc with the > standard C lib free(). That's not completly new. If Python is compiled in debug mode, you get a

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

On 04.02.2016 14:25, Victor Stinner wrote: > Thanks for your feedback, you are asking good questions :-) > > 2016-02-04 13:54 GMT+01:00 M.-A. Lemburg : >>> There are 536 calls to the functions PyMem_Malloc(), PyMem_Realloc() >>> and PyMem_Free(). >>> >>> I would prefer to modify

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

Thanks for your feedback, you are asking good questions :-) 2016-02-04 13:54 GMT+01:00 M.-A. Lemburg : >> There are 536 calls to the functions PyMem_Malloc(), PyMem_Realloc() >> and PyMem_Free(). >> >> I would prefer to modify a single place having to replace 536 calls :-/ > >

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

> There is an old discussion about the performance of PyMem_Malloc() memory > allocator. Oops, I forgot to mention that my patch is a follow-up of a previous patch showing nice speedup on dict: http://bugs.python.org/issue23601 (but I said it in my issue ;-)) Well, see

[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

Hi, There is an old discussion about the performance of PyMem_Malloc() memory allocator. CPython is stressing a lot memory allocators. Last time I made statistics, it was for the PEP 454: "For example, the Python test suites calls malloc() , realloc() or free() 270,000 times per second in