[Python-Dev] Re: PyGC and PyObject_Malloc introspection

2021-01-15 Thread Pablo Galindo Salgado
> Then the correct endpoint would more likely to be PyObject_New(), but > there's no way to intercept such calls for statistical analysis > currently. And as you wrote, if some code decide to use PyMalloc() > directly, then that memory won't be tracked. The one that CPython uses in debug mode to

[Python-Dev] Re: PyGC and PyObject_Malloc introspection

2021-01-15 Thread Julien Danjou
On Fri, Jan 15 2021, Pablo Galindo Salgado wrote: >> Exactly, which is a bit a bummer. Considering Python provides 3 >> different memory allocator, it'd be great if there was some ability to >> be sure that PyObject_Malloc pointer are actually PyObject, not >> Py_GC_HEAD. > > The allocators are

[Python-Dev] Re: PyGC and PyObject_Malloc introspection

2021-01-15 Thread Antoine Pitrou
Le 15/01/2021 à 13:08, Julien Danjou a écrit : > On Fri, Jan 15 2021, Antoine Pitrou wrote: > >> Also note that PyObject_Malloc() may also be used to allocate >> non-objects, for example a bytearray's payload, IIRC. > > Interesting. What's the rational for not using PyMem_Malloc() in such >

[Python-Dev] Re: PyGC and PyObject_Malloc introspection

2021-01-15 Thread Julien Danjou
On Fri, Jan 15 2021, Antoine Pitrou wrote: > Also note that PyObject_Malloc() may also be used to allocate > non-objects, for example a bytearray's payload, IIRC. Interesting. What's the rational for not using PyMem_Malloc() in such cases? -- Julien Danjou # Free Software hacker #

[Python-Dev] Re: PyGC and PyObject_Malloc introspection

2021-01-15 Thread Pablo Galindo Salgado
> Exactly, which is a bit a bummer. Considering Python provides 3 > different memory allocator, it'd be great if there was some ability to > be sure that PyObject_Malloc pointer are actually PyObject, not > Py_GC_HEAD. The allocators are specialized based on the allocation strategy and

[Python-Dev] Re: PyGC and PyObject_Malloc introspection

2021-01-15 Thread Antoine Pitrou
On Fri, 15 Jan 2021 09:36:11 +0100 Julien Danjou wrote: > On Thu, Jan 14 2021, Tim Peters wrote: > > > I'm not clear on exactly what it is you're after, but CPython faces > > the same question all the time: _given_ a pointer to an object, is > > there, or is there not, a GC header prepended?

[Python-Dev] Re: PyGC and PyObject_Malloc introspection

2021-01-15 Thread Julien Danjou
On Thu, Jan 14 2021, Tim Peters wrote: > I'm not clear on exactly what it is you're after, but CPython faces > the same question all the time: _given_ a pointer to an object, is > there, or is there not, a GC header prepended? That's answered by > this C API function: > > """ > int

[Python-Dev] Re: PyGC and PyObject_Malloc introspection

2021-01-15 Thread Julien Danjou
On Thu, Jan 14 2021, Pablo Galindo Salgado wrote: Hi Pablo, > Isn't this a similar problem that you have with regular malloc? When you > call malloc() with > some size, malloc actually will reserve more than that for > alignment purposes and for > bookkeeping and apart from some

[Python-Dev] Re: PyGC and PyObject_Malloc introspection

2021-01-14 Thread Tim Peters
[Julien Danjou] > ... > Supposedly PyObject_Malloc() returns some memory space to store a > PyObject. If that was true all the time, that would allow anyone to > introspect the allocated memory and understand why it's being used. > > Unfortunately, this is not the case. Objects whose types are

[Python-Dev] Re: PyGC and PyObject_Malloc introspection

2021-01-14 Thread Pablo Galindo Salgado
Hi Julien, Isn't this a similar problem that you have with regular malloc? When you call malloc() with some size, malloc actually will reserve more than that for alignment purposes and for bookkeeping and apart from some platform-specific APIs like malloc_usable_size() you cannot query that