[Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Matthew Paulson 

Hi:

I've spent some time performing memory leak analysis while using Python 
in an embedded configuration.


The pattern is:

   Py_Initialize();

   ... run empty python source file ...

   Py_Finalize();


I've identified several suspect areas including dictionary maitenace in 
import.c:~ 414


/* Clear the modules dict. */
PyDict_Clear(modules);
/* Restore the original builtins dict, to ensure that any
   user data gets cleared. */
dict = PyDict_Copy(interp->builtins);
if (dict == NULL)
PyErr_Clear();
PyDict_Clear(interp->builtins);
if (PyDict_Update(interp->builtins, interp->builtins_copy))
PyErr_Clear();
Py_XDECREF(dict);
/* Clear module dict copies stored in the interpreter state */


Is there someone in the group that would like to discuss this topic.  
There seems to be other leaks as well.  I'm new to Python-dev, but 
willing to help or work with someone who is more familiar with these 
areas than I.


Thanks,

Matt


--
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Brett Cannon 
Probably the best way to handle this, Matthew, is to open issues at
bugs.python.org for each of the leaks you have found and then they can be
discussed there.

And thanks for being willing to report these!

On Wed, 13 Jan 2016 at 11:42 Matthew Paulson  wrote:

> Hi:
>
> I've spent some time performing memory leak analysis while using Python in
> an embedded configuration.
>
> The pattern is:
>
> Py_Initialize();
>
> ... run empty python source file ...
>
> Py_Finalize();
>
>
> I've identified several suspect areas including dictionary maitenace in
> import.c:~ 414
>
> /* Clear the modules dict. */
> PyDict_Clear(modules);
> /* Restore the original builtins dict, to ensure that any
>user data gets cleared. */
> dict = PyDict_Copy(interp->builtins);
> if (dict == NULL)
> PyErr_Clear();
> PyDict_Clear(interp->builtins);
> if (PyDict_Update(interp->builtins, interp->builtins_copy))
> PyErr_Clear();
> Py_XDECREF(dict);
> /* Clear module dict copies stored in the interpreter state */
>
>
> Is there someone in the group that would like to discuss this topic.
> There seems to be other leaks as well.  I'm new to Python-dev, but willing
> to help or work with someone who is more familiar with these areas than I.
>
> Thanks,
>
> Matt
>
>
> --
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Modifying the self-signed.pythontest.net certificate

2016-01-13 Thread Martin Panter 
In order to fix the SSL test suite
, I would like to modify the
certificate used by https://self-signed.pythontest.net. So far I have
a patch ready for the pythontestdotnet repository, but I want to know
if I can just push to that repository, or if other steps are required.

Judging by , Georg Brandl was
involved in setting the server up. So far I haven’t heard anything
from him on the SSL tests bug, so I am asking here in case somebody
else can help.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Debugging using VS 2015

2016-01-13 Thread Brett Cannon 
Use PCbuild/get_externals.bat
 to
download the external dependencies.

On Wed, 13 Jan 2016 at 13:25 Eddy Quicksall  wrote:

> I am using 3.5.1. I’m adding an extension for my special case.
>
>
>
> I know this list is for development and that’s what I’m doing but I would
> like to use VS 2015 to do my debugging. If there is another list I should
> use to get the build to work then please let me know the list.
>
>
>
> I have been able to build using “PCbuild\build.bat –e” but I’m not able to
> build using PCbuild\pcbuild.sln. It gives lots of errors.
>
>
>
> I’m using Debug/Win32.
>
>
>
> Below is the output. Note that lots of .c files are missing …. Is there
> another package I should get (but then why would it build correctly with
> build.bat)? It could be that I need to add the equivalent to the “-e”
> option.
>
>
>
> 1>-- Rebuild All started: Project: pythoncore, Configuration: Debug
> Win32 --
>
> 2>-- Rebuild All started: Project: tcl, Configuration: Debug Win32
> --
>
> 3>-- Rebuild All started: Project: ssleay, Configuration: Debug Win32
> --
>
> 4>-- Rebuild All started: Project: sqlite3, Configuration: Debug Win32
> --
>
> 1>  Killing any running python_d.exe instances...
>
> 1>  'hg' is not recognized as an internal or external command,
>
> 1>  operable program or batch file.
>
> 1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(403,5): warning MSB3073: The
> command "hg id -b >
> "W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgbranch.txt"" exited
> with code 9009.
>
> 1>  'hg' is not recognized as an internal or external command,
>
> 1>  operable program or batch file.
>
> 1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(404,5): warning MSB3073: The
> command "hg id -i >
> "W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgversion.txt"" exited
> with code 9009.
>
> 1>  'hg' is not recognized as an internal or external command,
>
> 1>  operable program or batch file.
>
> 1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(405,5): warning MSB3073: The
> command "hg id -t >
> "W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgtag.txt"" exited
> with code 9009.
>
> 2>C:\Program Files
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(41,5):
> warning MSB8005: The property 'NMakeReBuildCommandLine' doesn't exist.
> Skipping...
>
> 5>-- Rebuild All started: Project: python3dll, Configuration: Debug
> Win32 --
>
> 1>  _bisectmodule.c
>
> 5>  python3dll.c
>
> 4>  sqlite3.c
>
> 3>  d1_both.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_both.c': No such file or
> directory
>
> 3>  d1_lib.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_lib.c': No such file or
> directory
>
> 1>  _codecsmodule.c
>
> 3>  d1_pkt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_pkt.c': No such file or
> directory
>
> 3>  d1_srtp.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_srtp.c': No such file or
> directory
>
> 3>  s2_clnt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_clnt.c': No such file or
> directory
>
> 3>  s2_enc.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_enc.c': No such file or
> directory
>
> 3>  s2_lib.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_lib.c': No such file or
> directory
>
> 3>  s2_meth.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_meth.c': No such file or
> directory
>
> 3>  s2_pkt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_pkt.c': No such file or
> directory
>
> 3>  s2_srvr.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_srvr.c': No such file or
> directory
>
> 3>  s23_clnt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_clnt.c': No such file or
> directory
>
> 3>  s23_lib.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_lib.c': No such file or
> directory
>
> 3>  s23_meth.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_meth.c': No such file or
> directory
>
> 3>  s23_pkt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_pkt.c': No such file or
> directory
>
> 3>  s23_srvr.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_srvr.c': No such file or
> directory
>
> 3>  s3_both.c
>
>

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Victor Stinner 
Hi,

2016-01-13 20:32 GMT+01:00 Matthew Paulson :
> I've spent some time performing memory leak analysis while using Python in an 
> embedded configuration.

Hum, did you try tracemalloc?

https://docs.python.org/dev/library/tracemalloc.html
https://pytracemalloc.readthedocs.org/

> Is there someone in the group that would like to discuss this topic.  There 
> seems to be other leaks as well.  I'm new to Python-dev, but willing to help 
> or work with someone who is more familiar with these areas than I.

Are you able to reproduce the leak with a simple program?

Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 510: Specialize functions with guards

2016-01-13 Thread Victor Stinner 
I extracted a patch from my FAT Python project to implement the PEP 510:
https://bugs.python.org/issue26098

FYI I also extracted the runtime part of the FAT Python optimizer and
put it on GitHub:
https://github.com/haypo/fat

The fat module provides specialize(), get_specialized() and
replace_consts() functions, but also 6 different guards: GuardArgType,
GuardBuiltins, GuardDict, GuardFunc, GuardGlobals, GuardTypeDict.

Victor
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Matthew Paulson 

Hi Victor:

No, I'm using the new heap analysis functions in DS2015.  We think we 
have found one issue. In the following sequence, dict has no side 
effects, yet it is used -- unless someone can shed light on why dict is 
used in this case:


/* Clear the modules dict. */
PyDict_Clear(modules);
/* Restore the original builtins dict, to ensure that any
   user data gets cleared. */
dict = PyDict_Copy(interp->builtins);
if (dict == NULL)
PyErr_Clear();
PyDict_Clear(interp->builtins);
if (PyDict_Update(interp->builtins, interp->builtins_copy))
PyErr_Clear();
Py_XDECREF(dict);

And removing dict from this sequence seems to have fixed one of the 
issues, yielding 14k per iteration.


Simple program: Good idea.  We will try that -- right now it's embedded 
in a more complex environment, but we have tried to strip it down to a 
very simple sequence.


The next item on our list is memory that is not getting freed after 
running simple string.  It's in the parsertok sequence -- it seems that 
the syntax tree is not getting cleared -- but this opinion is preliminary.


Best,

Matt

On 1/13/2016 5:10 PM, Victor Stinner wrote:

Hi,

2016-01-13 20:32 GMT+01:00 Matthew Paulson :

I've spent some time performing memory leak analysis while using Python in an 
embedded configuration.

Hum, did you try tracemalloc?

https://docs.python.org/dev/library/tracemalloc.html
https://pytracemalloc.readthedocs.org/


Is there someone in the group that would like to discuss this topic.  There 
seems to be other leaks as well.  I'm new to Python-dev, but willing to help or 
work with someone who is more familiar with these areas than I.

Are you able to reproduce the leak with a simple program?

Victor




--
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Andrew Barnert via Python-Dev 
On Jan 13, 2016, at 14:49, Matthew Paulson  wrote:
> 
> Hi Victor:
> 
> No, I'm using the new heap analysis functions in DS2015. 

Isn't that going to report any memory that Python's higher level allocators 
hold in their freelists as leaked, even though it isn't leaked?

> We think we have found one issue. In the following sequence, dict has no side 
> effects, yet it is used -- unless someone can shed light on why dict is used 
> in this case:

Where do you see an issue here? The dict will have one ref, so the decref at 
the end should return it to the freelist.

Also, it looks like there _is_ a side effect here. When you add a bunch of 
elements to a dict, it grows. When you delete a bunch of elements, it generally 
doesn't shrink. But when you clear the dict, it does shrink. So, copying it to 
a temporary dict, clearing it, updating it from the temporary dict, and then 
releasing the temporary dict should force it to shrink.

So, the overall effect should be that you have a smaller hash table for the 
builtins dict, and a chunk of memory sitting on the freelists ready to be 
reused. If your analyzer is showing the freelists as leaked, this will look 
like a net leak rather than a net recovery, but that's just a problem in the 
analyzer.

Of course I could be wrong, but I think the first step is to rule out the 
possibility that you're measuring the wrong thing...

> /* Clear the modules dict. */
> PyDict_Clear(modules);
> /* Restore the original builtins dict, to ensure that any
>user data gets cleared. */
> dict = PyDict_Copy(interp->builtins);
> if (dict == NULL)
> PyErr_Clear();
> PyDict_Clear(interp->builtins);
> if (PyDict_Update(interp->builtins, interp->builtins_copy))
> PyErr_Clear();
> Py_XDECREF(dict);
> 
> And removing dict from this sequence seems to have fixed one of the issues, 
> yielding 14k per iteration.

> Simple program: Good idea.  We will try that -- right now it's embedded in a 
> more complex environment, but we have tried to strip it down to a very simple 
> sequence.
> 
> The next item on our list is memory that is not getting freed after running 
> simple string.  It's in the parsertok sequence -- it seems that the syntax 
> tree is not getting cleared -- but this opinion is preliminary.
> 
> Best,
> 
> Matt
> 
>> On 1/13/2016 5:10 PM, Victor Stinner wrote:
>> Hi,
>> 
>> 2016-01-13 20:32 GMT+01:00 Matthew Paulson :
>>> I've spent some time performing memory leak analysis while using Python in 
>>> an embedded configuration.
>> Hum, did you try tracemalloc?
>> 
>> https://docs.python.org/dev/library/tracemalloc.html
>> https://pytracemalloc.readthedocs.org/
>> 
>>> Is there someone in the group that would like to discuss this topic.  There 
>>> seems to be other leaks as well.  I'm new to Python-dev, but willing to 
>>> help or work with someone who is more familiar with these areas than I.
>> Are you able to reproduce the leak with a simple program?
>> 
>> Victor
>> 
>> 
> 
> -- 
> 
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/abarnert%40yahoo.com
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Debugging using VS 2015

2016-01-13 Thread Brett Cannon 
Then your paths might be off; I would try with a fresh checkout and make
sure you can build Python cleanly before trying your embedded case to make
sure your tooling is set up (e.g., you don't have Mercurial installed so
that's a cause of one of your error messages).

On Wed, 13 Jan 2016 at 15:58 Eddy Quicksall  wrote:

> Those files already exist:
>
>
>
> W:\Python-3.5.1>PCbuild\get_externals.bat
>
> Fetching external libraries...
>
> bzip2-1.0.6 already exists, skipping.
>
> nasm-2.11.06 already exists, skipping.
>
> openssl-1.0.2d already exists, skipping.
>
> sqlite-3.8.11.0 already exists, skipping.
>
> tcl-core-8.6.4.2 already exists, skipping.
>
> tk-8.6.4.2 already exists, skipping.
>
> tix-8.4.3.6 already exists, skipping.
>
> xz-5.0.5 already exists, skipping.
>
> Finished.
>
>
>
> W:\Python-3.5.1>
>
>
>
> *From:* Brett Cannon [mailto:[email protected]]
> *Sent:* Wednesday, January 13, 2016 4:30 PM
> *To:* Eddy Quicksall; [email protected]
> *Subject:* Re: [Python-Dev] Debugging using VS 2015
>
>
>
> Use PCbuild/get_externals.bat
>  to
> download the external dependencies.
>
>
>
> On Wed, 13 Jan 2016 at 13:25 Eddy Quicksall  wrote:
>
> I am using 3.5.1. I’m adding an extension for my special case.
>
>
>
> I know this list is for development and that’s what I’m doing but I would
> like to use VS 2015 to do my debugging. If there is another list I should
> use to get the build to work then please let me know the list.
>
>
>
> I have been able to build using “PCbuild\build.bat –e” but I’m not able to
> build using PCbuild\pcbuild.sln. It gives lots of errors.
>
>
>
> I’m using Debug/Win32.
>
>
>
> Below is the output. Note that lots of .c files are missing …. Is there
> another package I should get (but then why would it build correctly with
> build.bat)? It could be that I need to add the equivalent to the “-e”
> option.
>
>
>
> 1>-- Rebuild All started: Project: pythoncore, Configuration: Debug
> Win32 --
>
> 2>-- Rebuild All started: Project: tcl, Configuration: Debug Win32
> --
>
> 3>-- Rebuild All started: Project: ssleay, Configuration: Debug Win32
> --
>
> 4>-- Rebuild All started: Project: sqlite3, Configuration: Debug Win32
> --
>
> 1>  Killing any running python_d.exe instances...
>
> 1>  'hg' is not recognized as an internal or external command,
>
> 1>  operable program or batch file.
>
> 1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(403,5): warning MSB3073: The
> command "hg id -b >
> "W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgbranch.txt"" exited
> with code 9009.
>
> 1>  'hg' is not recognized as an internal or external command,
>
> 1>  operable program or batch file.
>
> 1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(404,5): warning MSB3073: The
> command "hg id -i >
> "W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgversion.txt"" exited
> with code 9009.
>
> 1>  'hg' is not recognized as an internal or external command,
>
> 1>  operable program or batch file.
>
> 1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(405,5): warning MSB3073: The
> command "hg id -t >
> "W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgtag.txt"" exited
> with code 9009.
>
> 2>C:\Program Files
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(41,5):
> warning MSB8005: The property 'NMakeReBuildCommandLine' doesn't exist.
> Skipping...
>
> 5>-- Rebuild All started: Project: python3dll, Configuration: Debug
> Win32 --
>
> 1>  _bisectmodule.c
>
> 5>  python3dll.c
>
> 4>  sqlite3.c
>
> 3>  d1_both.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_both.c': No such file or
> directory
>
> 3>  d1_lib.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_lib.c': No such file or
> directory
>
> 1>  _codecsmodule.c
>
> 3>  d1_pkt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_pkt.c': No such file or
> directory
>
> 3>  d1_srtp.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_srtp.c': No such file or
> directory
>
> 3>  s2_clnt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_clnt.c': No such file or
> directory
>
> 3>  s2_enc.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_enc.c': No such file or
> directory
>
> 3>  s2_lib.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_lib.c': No such file or
> directory
>
> 3>  s2_meth.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_meth.c': No such file or
> directory
>
> 3>  s2_pkt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Matthew Paulson 

Hi Andrew:

These are all good points, and I defer to your experience -- I am new to 
python internals, but the fact remains that after multiple iterations of 
our embedded test case, we are seeing continued allocations (DS2015) and 
growth of the working set (windows task manager).  If your are pooling 
resources on the free list, wouldn't you expect these items to get 
reused and for things to stabilize after a while?  We're not seeing that.


I think Victor's suggestion of a very simple test case is probably the 
best idea.  I'll try to put that together in the next few days and if it 
also demonstrates the problem, then I'll submit it here.


Thanks for your time and help.

Best,

Matt

On 1/13/2016 6:45 PM, Andrew Barnert wrote:
On Jan 13, 2016, at 14:49, Matthew Paulson > wrote:



Hi Victor:

No, I'm using the new heap analysis functions in DS2015.


Isn't that going to report any memory that Python's higher level 
allocators hold in their freelists as leaked, even though it isn't leaked?


We think we have found one issue. In the following sequence, dict has 
no side effects, yet it is used -- unless someone can shed light on 
why dict is used in this case:


Where do you see an issue here? The dict will have one ref, so the 
decref at the end should return it to the freelist.


Also, it looks like there _is_ a side effect here. When you add a 
bunch of elements to a dict, it grows. When you delete a bunch of 
elements, it generally doesn't shrink. But when you clear the dict, it 
does shrink. So, copying it to a temporary dict, clearing it, updating 
it from the temporary dict, and then releasing the temporary dict 
should force it to shrink.


So, the overall effect should be that you have a smaller hash table 
for the builtins dict, and a chunk of memory sitting on the freelists 
ready to be reused. If your analyzer is showing the freelists as 
leaked, this will look like a net leak rather than a net recovery, but 
that's just a problem in the analyzer.


Of course I could be wrong, but I think the first step is to rule out 
the possibility that you're measuring the wrong thing...



/* Clear the modules dict. */
PyDict_Clear(modules);
/* Restore the original builtins dict, to ensure that any
   user data gets cleared. */
dict = PyDict_Copy(interp->builtins);
if (dict == NULL)
PyErr_Clear();
PyDict_Clear(interp->builtins);
if (PyDict_Update(interp->builtins, interp->builtins_copy))
PyErr_Clear();
Py_XDECREF(dict);

And removing dict from this sequence seems to have fixed one of the 
issues, yielding 14k per iteration.


Simple program: Good idea.  We will try that -- right now it's 
embedded in a more complex environment, but we have tried to strip it 
down to a very simple sequence.


The next item on our list is memory that is not getting freed after 
running simple string.  It's in the parsertok sequence -- it seems 
that the syntax tree is not getting cleared -- but this opinion is 
preliminary.


Best,

Matt

On 1/13/2016 5:10 PM, Victor Stinner wrote:

Hi,

2016-01-13 20:32 GMT+01:00 Matthew Paulson:

I've spent some time performing memory leak analysis while using Python in an 
embedded configuration.

Hum, did you try tracemalloc?

https://docs.python.org/dev/library/tracemalloc.html
https://pytracemalloc.readthedocs.org/


Is there someone in the group that would like to discuss this topic.  There 
seems to be other leaks as well.  I'm new to Python-dev, but willing to help or 
work with someone who is more familiar with these areas than I.

Are you able to reproduce the leak with a simple program?

Victor




--

___
Python-Dev mailing list
[email protected] 
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/abarnert%40yahoo.com


--
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Debugging using VS 2015

2016-01-13 Thread Steve Dower 

On 13Jan2016 1556, Eddy Quicksall wrote:

Those files already exist:

W:\Python-3.5.1>PCbuild\get_externals.bat
Fetching external libraries...
bzip2-1.0.6 already exists, skipping.
nasm-2.11.06 already exists, skipping.
openssl-1.0.2d already exists, skipping.
sqlite-3.8.11.0 already exists, skipping.
tcl-core-8.6.4.2 already exists, skipping.
tk-8.6.4.2 already exists, skipping.
tix-8.4.3.6 already exists, skipping.
xz-5.0.5 already exists, skipping.
Finished.
W:\Python-3.5.1>


You may need to delete the externals/openssl-1.0.2d directory and run 
get_externals.bat again.


Sometimes when you try to build without having downloaded the externals 
it creates enough directories that the script thinks it exists when it 
isn't all there.


Cheers,
Steve
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Nick Coghlan 
On 14 January 2016 at 10:18, Matthew Paulson  wrote:

> Hi Andrew:
>
> These are all good points, and I defer to your experience -- I am new to
> python internals, but the fact remains that after multiple iterations of
> our embedded test case, we are seeing continued allocations (DS2015) and
> growth of the working set (windows task manager).  If your are pooling
> resources on the free list, wouldn't you expect these items to get reused
> and for things to stabilize after a while?  We're not seeing that.
>
> I think Victor's suggestion of a very simple test case is probably the
> best idea.  I'll try to put that together in the next few days and if it
> also demonstrates the problem, then I'll submit it here.
>

If you want to throw your debugger at it, there's an existing
subinterpreter test case in _testembed that should exhibit any
Initialize/Finalize leaks:
https://github.com/python/cpython/blob/master/Programs/_testembed.c#L36

However, if there is one, our existing automated leak monitoring
unfortunately wouldn't pick it up, as the embedding tests run in a
subprocess rather than the main test process.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Trent Nelson 
Gflags/umdh is pretty useful on Windows, I used it to track down a few quirky 
PyParallel memory leaks.  Steps:


1.   Enable global flags: gflags –i python.exe +ust

2.   Launch Python.

3.   Enable the umdh tracer: umdh –p: -f:d1.log

4.   Kill it after a short run.

5.   Re-launch Python.

6.   Enable it again: umdh –p: -f:d2.log

7.   Let it run for longer – long enough to make sure it’s leaking memory.

8.   Kill it, then generate a dump file: umdh –d d1.log d2.log > dump.txt

(Those steps were pretty specific to my particular situation, but it should at 
least be a reasonable starting point for what to google to find out more.)

Here are two sample outputs that pin-pointed the exact leak path:


+   49116 (  49116 -  0)  6 allocsBackTrace9763CA0

+   6 (  6 -  0)BackTrace9763CA0allocations



ntdll!RtlpCallInterceptRoutine+40

ntdll!RtlAllocateHeap+79846

SQLSRV32!SQLAllocateMemory+26

markSQLSRV32!SQLAllocConnect+F6

SQLSRV32!SQLAllocHandle+83

ODBC32!RetcodeDriverInit+2D9

ODBC32!SQLInternalDriverConnectW+2F

ODBC32!CDispenser::CreateResource+DB

comsvcs!CHolder::SafeDispenserDriver::CreateResource+43

comsvcs!CHolder::AllocResource+24D

ODBC32!CDispenser::TryAllocResource+6E

ODBC32!CDispenser::GetActiveConnection+72

ODBC32!SQLDriverConnectW+9D4

pyodbc!Connect+14F 
(c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 85)

pyodbc!Connection_New+CD 
(c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 166)

pyodbc!mod_connect+579 
(c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\pyodbcmodule.cpp, 378)

python33!PyCFunction_Call+F3 
(c:\users\trent\home\src\pyparallel\objects\methodobject.c, 84)

python33!call_function+371 (c:\users\trent\home\src\pyparallel\python\ceval.c, 
4130)

python33!PyEval_EvalFrameEx+356C 
(c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)

python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 
4219)

python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 
4152)

python33!PyEval_EvalFrameEx+356C 
(c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)

python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 
4219)

python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 
4152)

python33!PyEval_EvalFrameEx+356C 
(c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)

python33!PyEval_EvalCodeEx+B4D 
(c:\users\trent\home\src\pyparallel\python\ceval.c, 3500)

python33!function_call+1BB 
(c:\users\trent\home\src\pyparallel\objects\funcobject.c, 639)

python33!PyObject_Call+7C 
(c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036)

python33!method_call+F9 
(c:\users\trent\home\src\pyparallel\objects\classobject.c, 353)

python33!PyObject_Call+7C 
(c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036)

python33!PyEval_CallObjectWithKeywords+16C 
(c:\users\trent\home\src\pyparallel\python\ceval.c, 4011)

python33!PxSocket_IOLoop+1249 
(c:\users\trent\home\src\pyparallel\python\pyparallel.c, 9128)



+   48432 (  48432 -  0)  6 allocsBackTrace97635E0

+   6 (  6 -  0)BackTrace97635E0allocations



ntdll!RtlpCallInterceptRoutine+40

ntdll!RtlAllocateHeap+79846

SQLSRV32!SQLAllocateMemory+26

SQLSRV32!SQLAllocConnect+4D

SQLSRV32!SQLAllocHandle+83

ODBC32!RetcodeDriverInit+2D9

ODBC32!SQLInternalDriverConnectW+2F

ODBC32!CDispenser::CreateResource+DB

comsvcs!CHolder::SafeDispenserDriver::CreateResource+43

comsvcs!CHolder::AllocResource+24D

ODBC32!CDispenser::TryAllocResource+6E

ODBC32!CDispenser::GetActiveConnection+72

ODBC32!SQLDriverConnectW+9D4

pyodbc!Connect+14F 
(c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 85)

pyodbc!Connection_New+CD 
(c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 166)

pyodbc!mod_connect+579 
(c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\pyodbcmodule.cpp, 378)

python33!PyCFunction_Call+F3 
(c:\users\trent\home\src\pyparallel\objects\methodobject.c, 84)

python33!call_function+371 (c:\users\trent\home\src\pyparallel\python\ceval.c, 
4130)

python33!PyEval_EvalFrameEx+356C 
(c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)

python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 
4219)

python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 
4152)

python33!PyEval_EvalFrameEx+356C 
(c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)

python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 
4219)

python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 
4152)

python33!PyEval_EvalFrameEx+356C 
(c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)

python33!PyEval_EvalCodeEx+B4D 
(c:\users\trent\home\src\pyparallel\python\ceval.c, 3500)

python33!function_call+1BB 
(c:\users\trent\home\src\pyparallel\objects\fu

Re: [Python-Dev] Modifying the self-signed.pythontest.net certificate

2016-01-13 Thread Benjamin Peterson 


On Wed, Jan 13, 2016, at 13:15, Martin Panter wrote:
> In order to fix the SSL test suite
> , I would like to modify the
> certificate used by https://self-signed.pythontest.net. So far I have
> a patch ready for the pythontestdotnet repository, but I want to know
> if I can just push to that repository, or if other steps are required.

It should suffice to update that repo. I suppose the server might have
to be kicked, too, as it's a cert change. Ping this thread if that
doesn't work.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-13 Thread Benjamin Peterson 
This is a "well-known" issue. Parts of the interpreter (and especially,
extension modules) cheerfully stash objects in global variables with no
way to clean them up. Fixing this is a large project, which probably
involves implementing PEP 489.

On Wed, Jan 13, 2016, at 11:32, Matthew Paulson wrote:
> Hi:
> 
> I've spent some time performing memory leak analysis while using Python 
> in an embedded configuration.
> 
> The pattern is:
> 
> Py_Initialize();
> 
> ... run empty python source file ...
> 
> Py_Finalize();
> 
> 
> I've identified several suspect areas including dictionary maitenace in 
> import.c:~ 414
> 
>  /* Clear the modules dict. */
>  PyDict_Clear(modules);
>  /* Restore the original builtins dict, to ensure that any
> user data gets cleared. */
>  dict = PyDict_Copy(interp->builtins);
>  if (dict == NULL)
>  PyErr_Clear();
>  PyDict_Clear(interp->builtins);
>  if (PyDict_Update(interp->builtins, interp->builtins_copy))
>  PyErr_Clear();
>  Py_XDECREF(dict);
>  /* Clear module dict copies stored in the interpreter state */
> 
> 
> Is there someone in the group that would like to discuss this topic.  
> There seems to be other leaks as well.  I'm new to Python-dev, but 
> willing to help or work with someone who is more familiar with these 
> areas than I.
> 
> Thanks,
> 
> Matt
> 
> 
> -- 
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
> Email had 1 attachment:
> + MattSig.JPG
>   52k (image/jpeg)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Modifying the self-signed.pythontest.net certificate

2016-01-13 Thread Martin Panter 
On 14 January 2016 at 05:34, Benjamin Peterson  wrote:
> On Wed, Jan 13, 2016, at 13:15, Martin Panter wrote:
>> In order to fix the SSL test suite
>> , I would like to modify the
>> certificate used by https://self-signed.pythontest.net. So far I have
>> a patch ready for the pythontestdotnet repository, but I want to know
>> if I can just push to that repository, or if other steps are required.
>
> It should suffice to update that repo. I suppose the server might have
> to be kicked, too, as it's a cert change. Ping this thread if that
> doesn't work.

It seems to have worked, thanks. Not immediately, but after waiting
~20 minutes the certificate is now updated. I am working on an update
for all branches 3.2+ and 2.7; I expect test_httplib will be broken
until I fix it.

BTW  is giving me a 403 Forbidden
response, but it was doing this before I pushed my change, so I think
it’s not my fault :)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Modifying the self-signed.pythontest.net certificate

2016-01-13 Thread Benjamin Peterson 


On Wed, Jan 13, 2016, at 23:31, Martin Panter wrote:
> On 14 January 2016 at 05:34, Benjamin Peterson 
> wrote:
> > On Wed, Jan 13, 2016, at 13:15, Martin Panter wrote:
> >> In order to fix the SSL test suite
> >> , I would like to modify the
> >> certificate used by https://self-signed.pythontest.net. So far I have
> >> a patch ready for the pythontestdotnet repository, but I want to know
> >> if I can just push to that repository, or if other steps are required.
> >
> > It should suffice to update that repo. I suppose the server might have
> > to be kicked, too, as it's a cert change. Ping this thread if that
> > doesn't work.
> 
> It seems to have worked, thanks. Not immediately, but after waiting
> ~20 minutes the certificate is now updated. I am working on an update
> for all branches 3.2+ and 2.7; I expect test_httplib will be broken
> until I fix it.

Thank you for taking care of this issue.

> 
> BTW  is giving me a 403 Forbidden
> response, but it was doing this before I pushed my change, so I think
> it’s not my fault :)

Fixed.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com