Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-14 Thread Nick Coghlan
On 15 January 2016 at 05:25, Matthew Paulson wrote: > Hi All: > > I've created a simple program to make sure I wasn't lying to you all ;-> > > Here it is: > > for (ii = 0; ii < 100; ii++) > { > Py_Initialize(); > > if ((code = Py_CompileString(p, "foo", Py_file_inp

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-14 Thread Matthew Paulson
Hi All: I've created a simple program to make sure I wasn't lying to you all ;-> Here it is: for (ii = 0; ii < 100; ii++) { Py_Initialize(); if ((code = Py_CompileString(p, "foo", Py_file_input)) == NULL) printf("PyRun_SimpleString() failed\n");

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-14 Thread Stefan Krah
Petr Viktorin gmail.com> writes: > The hairy details on why the global variables haven't yet gone away are > on import-sig [0]. Nick suggested a workable solution there that I > really need to go back to and implement. > > [0] https://mail.python.org/pipermail/import-sig/2015-July/001022.html I

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-14 Thread Petr Viktorin
On 01/14/2016 10:45 AM, Nick Coghlan wrote: > On 14 January 2016 at 15:42, Benjamin Peterson wrote: >> This is a "well-known" issue. Parts of the interpreter (and especially, >> extension modules) cheerfully stash objects in global variables with no >> way to clean them up. Fixing this is a large

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-14 Thread Nick Coghlan
On 14 January 2016 at 15:42, Benjamin Peterson wrote: > This is a "well-known" issue. Parts of the interpreter (and especially, > extension modules) cheerfully stash objects in global variables with no > way to clean them up. Fixing this is a large project, which probably > involves implementing P

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Benjamin Peterson
This is a "well-known" issue. Parts of the interpreter (and especially, extension modules) cheerfully stash objects in global variables with no way to clean them up. Fixing this is a large project, which probably involves implementing PEP 489. On Wed, Jan 13, 2016, at 11:32, Matthew Paulson wrote:

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Trent Nelson
ers Subject: Re: [Python-Dev] Discussion related to memory leaks requested Hi Andrew: These are all good points, and I defer to your experience -- I am new to python internals, but the fact remains that after multiple iterations of our embedded test case, we are seeing continued allocations (DS2

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Nick Coghlan
On 14 January 2016 at 10:18, Matthew Paulson wrote: > Hi Andrew: > > These are all good points, and I defer to your experience -- I am new to > python internals, but the fact remains that after multiple iterations of > our embedded test case, we are seeing continued allocations (DS2015) and > gro

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Matthew Paulson
Hi Andrew: These are all good points, and I defer to your experience -- I am new to python internals, but the fact remains that after multiple iterations of our embedded test case, we are seeing continued allocations (DS2015) and growth of the working set (windows task manager). If your are p

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Andrew Barnert via Python-Dev
On Jan 13, 2016, at 14:49, Matthew Paulson wrote: > > Hi Victor: > > No, I'm using the new heap analysis functions in DS2015. Isn't that going to report any memory that Python's higher level allocators hold in their freelists as leaked, even though it isn't leaked? > We think we have found o

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Matthew Paulson
Hi Victor: No, I'm using the new heap analysis functions in DS2015. We think we have found one issue. In the following sequence, dict has no side effects, yet it is used -- unless someone can shed light on why dict is used in this case: /* Clear the modules dict. */ PyDict_Clear(modules

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Victor Stinner
Hi, 2016-01-13 20:32 GMT+01:00 Matthew Paulson : > I've spent some time performing memory leak analysis while using Python in an > embedded configuration. Hum, did you try tracemalloc? https://docs.python.org/dev/library/tracemalloc.html https://pytracemalloc.readthedocs.org/ > Is there someon

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Brett Cannon
Probably the best way to handle this, Matthew, is to open issues at bugs.python.org for each of the leaks you have found and then they can be discussed there. And thanks for being willing to report these! On Wed, 13 Jan 2016 at 11:42 Matthew Paulson wrote: > Hi: > > I've spent some time perform

[Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Matthew Paulson
Hi: I've spent some time performing memory leak analysis while using Python in an embedded configuration. The pattern is: Py_Initialize(); ... run empty python source file ... Py_Finalize(); I've identified several suspect areas including dictionary maitenace in import.c:~ 414