Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On Tue, Apr 16, 2019 at 2:11 AM Victor Stinner wrote: > Hi Michael, > > Do you know the tracemalloc module? Did you try it? It works on a > regular Python (compiled in debug mode). > > I would be curious to know if tracemalloc also allows you to track the > memory leak. > > Playing around with

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

Since Python 3.6, you can use PYTHONMALLOC=malloc for Valgrind: it avoids false alarms produced by the pymalloc allocator. Victor Le mar. 16 avr. 2019 à 12:09, Nathaniel Smith a écrit : > > On Mon, Apr 15, 2019 at 8:58 PM Michael Sullivan wrote: > > > > On Mon, Apr 15, 2019 at 4:06 PM

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On Mon, Apr 15, 2019 at 8:58 PM Michael Sullivan wrote: > > On Mon, Apr 15, 2019 at 4:06 PM Nathaniel Smith wrote: >> >> On Mon, Apr 15, 2019, 15:27 Michael Sullivan wrote: >>> >>> > The main question is if anyone ever used Py_TRACE_REFS? Does someone >>> > use sys.getobjects() or

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

Hi Michael, Do you know the tracemalloc module? Did you try it? It works on a regular Python (compiled in debug mode). I would be curious to know if tracemalloc also allows you to track the memory leak. sys.getobjects() is just a list of objects. Do you have a tool written on top of it to track

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On Mon, Apr 15, 2019 at 4:06 PM Nathaniel Smith wrote: > On Mon, Apr 15, 2019, 15:27 Michael Sullivan wrote: > >> > The main question is if anyone ever used Py_TRACE_REFS? Does someone >> > use sys.getobjects() or PYTHONDUMPREFS environment variable? >> >> I used sys.getobjects() today to track

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On Mon, Apr 15, 2019, 15:27 Michael Sullivan wrote: > > The main question is if anyone ever used Py_TRACE_REFS? Does someone > > use sys.getobjects() or PYTHONDUMPREFS environment variable? > > I used sys.getobjects() today to track down a memory leak in the > mypyc-compiled version of mypy. > >

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

> The main question is if anyone ever used Py_TRACE_REFS? Does someone > use sys.getobjects() or PYTHONDUMPREFS environment variable? I used sys.getobjects() today to track down a memory leak in the mypyc-compiled version of mypy. We were leaking memory badly but no sign of the leak was showing

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On Mon, 15 Apr 2019 12:50:00 +0200 Antoine Pitrou wrote: > On Thu, 11 Apr 2019 08:26:47 -0700 > Steve Dower wrote: > > On 10Apr2019 1917, Nathaniel Smith wrote: > > > It sounds like --with-pydebug has accumulated a big grab bag of > > > unrelated features, mostly stuff that was useful at some

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On 12Apr2019 1819, Nathaniel Smith wrote: On Fri, Apr 12, 2019 at 5:05 PM Steve Dower wrote: On 12Apr.2019 1643, Nathaniel Smith wrote: On Thu, Apr 11, 2019 at 8:26 AM Steve Dower wrote: The very first question I asked was whether this would let us converge the ABIs, and the answer was

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On Thu, 11 Apr 2019 08:26:47 -0700 Steve Dower wrote: > On 10Apr2019 1917, Nathaniel Smith wrote: > > It sounds like --with-pydebug has accumulated a big grab bag of > > unrelated features, mostly stuff that was useful at some point for > > some CPython dev trying to debug CPython itself? It's

[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

>> The very first question I asked was whether this would let us converge >> the ABIs, and the answer was "no". The answer is yes and it's my primary goal. See my first email: "This change makes the debug build ABI closer to the release build ABI". To be honest, I am now lost in this long

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On Fri, Apr 12, 2019 at 5:05 PM Steve Dower wrote: > > On 12Apr.2019 1643, Nathaniel Smith wrote: > > On Thu, Apr 11, 2019 at 8:26 AM Steve Dower wrote: > >> > >> On 10Apr2019 1917, Nathaniel Smith wrote: > > I don't know how many people use Py_TRACE_REFS, but if we can't find > > anyone on

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On 12Apr.2019 1643, Nathaniel Smith wrote: > On Thu, Apr 11, 2019 at 8:26 AM Steve Dower wrote: >> >> On 10Apr2019 1917, Nathaniel Smith wrote: > I don't know how many people use Py_TRACE_REFS, but if we can't find > anyone on python-dev who uses it then it must be pretty rare. If > dropping

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On Thu, Apr 11, 2019 at 8:26 AM Steve Dower wrote: > > On 10Apr2019 1917, Nathaniel Smith wrote: > > It sounds like --with-pydebug has accumulated a big grab bag of > > unrelated features, mostly stuff that was useful at some point for > > some CPython dev trying to debug CPython itself? It's

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

Le sam. 13 avr. 2019 à 00:38, Greg Ewing a écrit : > Rather than removing features altogether, maybe the debug > build could be split into a number of separate features > that can be enabled individually? I don't propose to *remove* a feature, but just to *disable* it *by default* (when Python

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

Victor Stinner wrote: I'm not sure of what you means by "objects placed at static memory": the double linked list of all Python objects is created at runtime. _ob_next and _ob_prev are initialized statically to NULL. The trick of allocating extra memory in front of the object would be harder

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

Victor Stinner wrote: Python is used on devices with low memory (ex: 256 MiB for the whole system). Allowing developers to use a debug build on such devices seem to be a legit rationale for such change. Rather than removing features altogether, maybe the debug build could be split into a

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

12.04.19 17:40, Guido van Rossum пише: So let's leave it off by default even in debug builds. But let's not delete the macros. Maybe switch it on (together with other disabled by default options) on some fast buildbot? ___ Python-Dev mailing list

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On Fri, Apr 12, 2019 at 5:51 AM Stefan Behnel wrote: > Serhiy Storchaka schrieb am 11.04.19 um 17:30: > > If reducing the Python memory footprint is an argument for disabling > > Py_TRACE_REFS, it is a weak argument because there is larger overhead in > > the debug build. > > I think what Victor

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

Serhiy Storchaka schrieb am 11.04.19 um 17:30: > If reducing the Python memory footprint is an argument for disabling > Py_TRACE_REFS, it is a weak argument because there is larger overhead in > the debug build. I think what Victor is argueing is rather that we have better ways to debug memory

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

Le ven. 12 avr. 2019 à 12:57, Victor Stinner a écrit : > I wrote https://bugs.python.org/issue36611 to remove the serialno field > of debug hooks on Python memory allocators: it reduces > the memory footprint by 5% (ex: 1.2 MiB on 33.0 MiB when running > test_asyncio). I measured the memory

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

Le jeu. 11 avr. 2019 à 17:33, Serhiy Storchaka a écrit : > If reducing the Python memory footprint is an argument for disabling > Py_TRACE_REFS, it is a weak argument because there is larger overhead in > the debug build. The "serialno" field of debug memory allocators is documented as: "an

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On Thu, Apr 11, 2019 at 8:32 AM Serhiy Storchaka wrote: > On other hand, since using the debug allocator doesn't cause problems > with compatibility, it may be possible to use similar technique for the > objects double list. Although this is not easy because of objects placed > at static memory.

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

11.04.19 12:28, Victor Stinner пише: Le jeu. 11 avr. 2019 à 07:49, Serhiy Storchaka a écrit : 10.04.19 14:01, Victor Stinner пише: Disabling Py_TRACE_REFS by default in debug mode reduces the Python memory footprint. Py_TRACE_REFS costs 2 pointers per PyObject: 16 bytes on 64-bit platforms.

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On 10Apr2019 1917, Nathaniel Smith wrote: It sounds like --with-pydebug has accumulated a big grab bag of unrelated features, mostly stuff that was useful at some point for some CPython dev trying to debug CPython itself? It's clearly not designed with end users as the primary audience, given

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On 11Apr2019 0228, Victor Stinner wrote: Le jeu. 11 avr. 2019 à 07:49, Serhiy Storchaka a écrit : 10.04.19 14:01, Victor Stinner пише: Disabling Py_TRACE_REFS by default in debug mode reduces the Python memory footprint. Py_TRACE_REFS costs 2 pointers per PyObject: 16 bytes on 64-bit

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

Le jeu. 11 avr. 2019 à 07:49, Serhiy Storchaka a écrit : > 10.04.19 14:01, Victor Stinner пише: > > Disabling Py_TRACE_REFS by default in debug mode reduces the Python > > memory footprint. Py_TRACE_REFS costs 2 pointers per PyObject: 16 > > bytes on 64-bit platforms. > > Does not the memory

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

10.04.19 14:01, Victor Stinner пише: Disabling Py_TRACE_REFS by default in debug mode reduces the Python memory footprint. Py_TRACE_REFS costs 2 pointers per PyObject: 16 bytes on 64-bit platforms. Does not the memory allocator in debug mode have even larger cost per allocated block?

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On Wed, Apr 10, 2019 at 1:50 PM Steve Dower wrote: > > On 10Apr2019 1227, Nathaniel Smith wrote: > > On Wed, Apr 10, 2019, 04:04 Victor Stinner > > wrote: > > I don't think that I ever used sys.getobjects(), whereas many projects > > use gc.get_objects() which

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

Le mer. 10 avr. 2019 à 21:45, Brett Cannon a écrit : >> Can anyone explain what pydebug builds are... for? Confession: I've never >> used them myself, and don't know why I would want to. > > There is a bunch of extra things done in a debug build, e.g. all freed memory > is blanked out with a

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

Le mer. 10 avr. 2019 à 20:09, Steve Dower a écrit : > > The main question is if anyone ever used Py_TRACE_REFS? Does someone > > use sys.getobjects() or PYTHONDUMPREFS environment variable? > > > > Using PYTHONDUMPREFS=1 on a debug build (with Py_TRACE_REFS) does > > simply crash Python 3.7 at

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On 4/10/2019 2:45 PM, Steve Dower wrote: It's debug mode, and so you should expect less efficient memory and CPU usage. On my Windows machine, 'python -m test -ugui' takes about twice as long. That's why we have two modes - so that it's easier to debug issues. -- Terry Jan Reedy

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On 10Apr2019 1227, Nathaniel Smith wrote: On Wed, Apr 10, 2019, 04:04 Victor Stinner > wrote: I don't think that I ever used sys.getobjects(), whereas many projects use gc.get_objects() which is also available in release builds (not only in debug builds).

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On Wed, Apr 10, 2019 at 12:30 PM Nathaniel Smith wrote: > On Wed, Apr 10, 2019, 04:04 Victor Stinner wrote: > >> Le mar. 9 avr. 2019 à 22:16, Steve Dower a >> écrit : >> > What are the other changes that would be required? >> >> I don't know. >> >> > And is there another >> > way to get the

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On Wed, Apr 10, 2019, 04:04 Victor Stinner wrote: > Le mar. 9 avr. 2019 à 22:16, Steve Dower a écrit > : > > What are the other changes that would be required? > > I don't know. > > > And is there another > > way to get the same functionality without ABI modifications? > > Py_TRACE_REFS is a

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

I recall finding memory leaks using this. (E.g. I remember a leak in Zope due to a cache that was never pruned.) But presumably gc.get_objects() would have been sufficient. (IIRC it didn't exist at the time.) On Wed, Apr 10, 2019 at 11:48 AM Steve Dower wrote: > On 10Apr2019 1109, Steve Dower

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On 10Apr2019 1109, Steve Dower wrote: On 10Apr2019 0401, Victor Stinner wrote: I think it's worthwhile if we can really get to debug and non-debug builds being ABI compatible. Getting partway there in this case doesn't seem to offer any benefits. Disabling Py_TRACE_REFS by default in debug

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On 10Apr2019 0401, Victor Stinner wrote: Le mar. 9 avr. 2019 à 22:16, Steve Dower a écrit : What are the other changes that would be required? I don't know. And is there another way to get the same functionality without ABI modifications? Py_TRACE_REFS is a double linked list of *all*

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

Le mar. 9 avr. 2019 à 22:16, Steve Dower a écrit : > What are the other changes that would be required? I don't know. > And is there another > way to get the same functionality without ABI modifications? Py_TRACE_REFS is a double linked list of *all* Python objects. To get this functionality,

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

On 09Apr2019 0925, Victor Stinner wrote: This change makes the debug build ABI closer to the release build ABI, but I am not sure how to compare these two ABI. Technically, C extensions still need to be recompiled. What do you think? What are the other changes that would be required? And is

[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

Hi, When Python is built in debug mode, PyObject gets 2 new fields: _ob_prev and _ob_next. These fields change the offset of following fields in the PyObject structure and so breaks the ABI. I propose to modify the debug build (Py_DEBUG) to no longer imply Py_TRACE_REFS. Antoine Pitrou proposed