Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-14 Thread Nick Coghlan 
On 14 January 2016 at 15:42, Benjamin Peterson  wrote:
> This is a "well-known" issue. Parts of the interpreter (and especially,
> extension modules) cheerfully stash objects in global variables with no
> way to clean them up. Fixing this is a large project, which probably
> involves implementing PEP 489.

The actual multi-phase extension module import system from 489 was
implemented for 3.5, but indeed, the modules with stashed global state
haven't been converted yet.

I didn't think we loaded any of those by default, though...

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-14 Thread Petr Viktorin 
On 01/14/2016 10:45 AM, Nick Coghlan wrote:
> On 14 January 2016 at 15:42, Benjamin Peterson  wrote:
>> This is a "well-known" issue. Parts of the interpreter (and especially,
>> extension modules) cheerfully stash objects in global variables with no
>> way to clean them up. Fixing this is a large project, which probably
>> involves implementing PEP 489.
> 
> The actual multi-phase extension module import system from 489 was
> implemented for 3.5, but indeed, the modules with stashed global state
> haven't been converted yet.

The hairy details on why the global variables haven't yet gone away are
on import-sig [0]. Nick suggested a workable solution there that I
really need to go back to and implement.

[0] https://mail.python.org/pipermail/import-sig/2015-July/001022.html



___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-14 Thread Stefan Krah 
Petr Viktorin  gmail.com> writes:
> The hairy details on why the global variables haven't yet gone away are
> on import-sig [0]. Nick suggested a workable solution there that I
> really need to go back to and implement.
> 
> [0] https://mail.python.org/pipermail/import-sig/2015-July/001022.html

I want to add here that existing schemes for eliminating global variables
are inefficient (20% speed hit for _decimal), so a complete solution
would have to address that as well.


Stefan Krah

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 506 secrets module

2016-01-14 Thread Brett Cannon 
I noticed an article about default rand usage in Go

from the Go Weekly newsletter and it reminded me about PEP 506 and the
secrets module. That's when I noticed that the PEP is still open. What is
the current blocker on the PEP?

On Thu, 15 Oct 2015 at 17:57 Steven D'Aprano  wrote:

> Hi,
>
> As extensively discussed on Python-Ideas, the secrets module and PEP 506
> is (I hope) ready for pronouncement.
>
> https://www.python.org/dev/peps/pep-0506/
>
> There is code and tests here:
>
> https://bitbucket.org/sdaprano/secrets
>
>
> or you can run
>
> hg clone https://[email protected]/sdaprano/secrets
>
>
> The code is written for and tested on Python 2.6, 2.7, 3.1 - 3.4.
>
>
>
> --
> Steve
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 506 secrets module

2016-01-14 Thread Guido van Rossum 
I think the discussion petered out and nobody asked me to approve it yet
(or I lost track of it). I'm almost happy to approve it in the current
state. My only quibble is with some naming -- I'm not sure that a
super-generic name like 'equal' is better than the original
('compare_digest'), and I would have picked a different name for token_url
-- probably token_urlsafe. But maybe Steven can convince me that the names
currently in the PEP are better. (I also don't like the wishy-washy
position of the PEP on the actual specs of the proposed functions. But I'm
fine with the actual implementation shown as the spec.)

On Thu, Jan 14, 2016 at 10:36 AM, Brett Cannon  wrote:

> I noticed an article about default rand usage in Go
> 
> from the Go Weekly newsletter and it reminded me about PEP 506 and the
> secrets module. That's when I noticed that the PEP is still open. What is
> the current blocker on the PEP?
>
> On Thu, 15 Oct 2015 at 17:57 Steven D'Aprano  wrote:
>
>> Hi,
>>
>> As extensively discussed on Python-Ideas, the secrets module and PEP 506
>> is (I hope) ready for pronouncement.
>>
>> https://www.python.org/dev/peps/pep-0506/
>>
>> There is code and tests here:
>>
>> https://bitbucket.org/sdaprano/secrets
>>
>>
>> or you can run
>>
>> hg clone https://[email protected]/sdaprano/secrets
>>
>>
>> The code is written for and tested on Python 2.6, 2.7, 3.1 - 3.4.
>>
>>
>>
>> --
>> Steve
>> ___
>> Python-Dev mailing list
>> [email protected]
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
>
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-14 Thread Matthew Paulson 

Hi All:

I've created a simple program to make sure I wasn't lying to you all ;->

Here it is:

for (ii = 0; ii < 100; ii++)
{
Py_Initialize();

if ((code = Py_CompileString(p, "foo", Py_file_input)) == NULL)
printf("PyRun_SimpleString() failed\n");
else
{
if (PyRun_SimpleString(p) == -1)
printf("PyRun_SimpleString() failed\n");

Py_CLEAR(code);
}

Py_Finalize();
}

This sequence causes about 10k growth per iteration and after many 
cycles, there's no indication that any pooling logic is helping. Our 
"useful" example is slightly more complex, and therefore may explain why 
I was seeing about 16k per iteration.


Unless I've done something obviously wrong, I tend to believe Benjamin's 
claim that this issue is well known.


Suggestion: I have had great success with similar problems in the past 
by using a pools implementation sitting on top of what I call a "block 
memory allocator".   The bottom (block) allocator grabs large blocks 
from the heap and then doles them out to the pools layer, which in turn 
doles them out to the requester.  When client memory is freed -- it is 
NOT -- rather it's added to the pool which contains like-sized blocks -- 
call it an "organized free list". This is a very, very fast way to 
handle high allocation frequency patterns.  Finally, during shutdown, 
the pool simply vaporizes and the block allocator returns a the (fewer) 
large blocks back to the heap.  This avoids thrashing the heap, forcing 
it to coalesce inefficiently and also avoids heap fragmentation, which 
can cause unwanted growth as well...


Note that this would be a "hard-reset" of all allocated memory, and any 
global data in the text segment would also have to be cleared, but it 
would provide a fast, clean way to ensure that each invocation was 100% 
clean.


I don't claim to understand all the intricacies of the many way python 
can be embedded, but as I said, this strategy has worked very well for 
me in the past building servers written in C that have to stay up for 
months at a time.


Happy to discuss further, if anyone has any interest.

Best,

Matt




On 1/14/2016 4:45 AM, Nick Coghlan wrote:

On 14 January 2016 at 15:42, Benjamin Peterson  wrote:

This is a "well-known" issue. Parts of the interpreter (and especially,
extension modules) cheerfully stash objects in global variables with no
way to clean them up. Fixing this is a large project, which probably
involves implementing PEP 489.

The actual multi-phase extension module import system from 489 was
implemented for 3.5, but indeed, the modules with stashed global state
haven't been converted yet.

I didn't think we loaded any of those by default, though...

Cheers,
Nick.



--
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 509

2016-01-14 Thread Stephen J. Turnbull 
Terry Reedy writes:

 > While I understand the rationale against __version__, it strikes me as a 
 > better description of what it is, and easier on the brain than 
 > __cache_token__.  Maybe there is something even better, such as 
 > __seqnum__.

Or __generation__, as in "generational garbage collector"?

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Discussion related to memory leaks requested

2016-01-14 Thread Nick Coghlan 
On 15 January 2016 at 05:25, Matthew Paulson  wrote:

> Hi All:
>
> I've created a simple program to make sure I wasn't lying to you all ;->
>
> Here it is:
>
> for (ii = 0; ii < 100; ii++)
> {
> Py_Initialize();
>
> if ((code = Py_CompileString(p, "foo", Py_file_input)) == NULL)
> printf("PyRun_SimpleString() failed\n");
> else
> {
> if (PyRun_SimpleString(p) == -1)
> printf("PyRun_SimpleString() failed\n");
>
> Py_CLEAR(code);
> }
>
> Py_Finalize();
> }
>
> This sequence causes about 10k growth per iteration and after many cycles,
> there's no indication that any pooling logic is helping.  Our "useful"
> example is slightly more complex, and therefore may explain why I was
> seeing about 16k per iteration.
>
> Unless I've done something obviously wrong, I tend to believe Benjamin's
> claim that this issue is well known.
>
> Suggestion: I have had great success with similar problems in the past by
> using a pools implementation sitting on top of what I call a "block memory
> allocator".   The bottom (block) allocator grabs large blocks from the heap
> and then doles them out to the pools layer, which in turn doles them out to
> the requester.  When client memory is freed -- it is NOT -- rather it's
> added to the pool which contains like-sized blocks -- call it an "organized
> free list". This is a very, very fast way to handle high allocation
> frequency patterns.  Finally, during shutdown, the pool simply vaporizes
> and the block allocator returns a the (fewer) large blocks back to the
> heap.  This avoids thrashing the heap, forcing it to coalesce inefficiently
> and also avoids heap fragmentation, which can cause unwanted growth as
> well...
>
> Note that this would be a "hard-reset" of all allocated memory, and any
> global data in the text segment would also have to be cleared, but it would
> provide a fast, clean way to ensure that each invocation was 100% clean.
>

CPython does use an arena based allocator, but PyFinalize doesn't purge it
(if it did, there'd be segfaults rather than memory growth when modules
keep pointers across Initialize/Finalize cycles).

Building with PYMALLOC_DEBUG and setting PYTHONMALLOCSTATS in the
environment will cause it to dump debugging info during Py_Finalize.

Building with Py_TRACE_REFS and setting PYTHONDUMPREFS also provides info
on live Python objects during shutdown.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com