On 04.02.2016 14:25, Victor Stinner wrote: > Thanks for your feedback, you are asking good questions :-) > > 2016-02-04 13:54 GMT+01:00 M.-A. Lemburg <m...@egenix.com>: >>> There are 536 calls to the functions PyMem_Malloc(), PyMem_Realloc() >>> and PyMem_Free(). >>> >>> I would prefer to modify a single place having to replace 536 calls :-/ >> >> You have a point there, but I don't think it'll work out >> that easily, since we are using such calls to e.g. pass >> dynamically allocated buffers to code in extensions (which then >> have to free the buffers again). > > Ah, interesting. But I'm not sure that we delegate the responsability > of freeing the memory to external libraries. Usually, it's more the > opposite: a library gives us an allocated memory block, and we have to > free it. No?
Sometimes, yes, but we also do allocations for e.g. parsing values in Python argument tuples (e.g. using "es" or "et"): https://docs.python.org/3.6/c-api/arg.html We do document to use PyMem_Free() on those; not sure whether everyone does this though. > I checked if we call directly malloc() to pass the buffer to a > library, but I failed to find such case. > > Again, in debug mode, calling free() on a memory block allocated by > PyMem_Malloc() will likely crash. Since we run the Python test suite > with a Python compiled in debug mode, we would already have detected > such bug, no? The Python test suite doesn't test Python C extensions, so it's not surprising that it passes :-) > See also my old issue http://bugs.python.org/issue18203 which replaced > almost all direct calls to malloc() with PyMem_Malloc() or > PyMem_RawMalloc(). > >> Good question. I guess developers simply thought of PyObject_Malloc() >> being for PyObjects, > > Yeah, I also understood that, but in practice, it looks like > PyMem_Malloc() is slower than so using it makes the code less > efficient than it can be. > > Instead of teaching developers that well, in fact, PyObject_Malloc() > is unrelated to object programming, I think that it's simpler to > modify PyMem_Malloc() to reuse pymalloc ;-) Perhaps if you add some guards somewhere :-) Seriously, this may work if C extensions use the APIs consistently, but in order to tell, we'd need to check few. I know that I switched over all mx Extensions to use PyObject_*() instead of PyMem_*() or native malloc() several years ago and have not run into any issues. I guess the main question then is whether pymalloc is good enough for general memory allocation needs; and the answer may well be "yes". BTW: Tuning pymalloc for commonly used object sizes is another area where Python could gain better performance, i.e. reserve more / pre-allocate space for often used block sizes. pymalloc will also only work well for small blocks (up to 512 bytes). Everything else is routed to the system malloc(). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 04 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com