[Python-Dev] Inconsistent nesting of scopes in exec(..., locals())

2010-04-23 Thread Joachim B Haga
There seem to be an inconsistency in the handling of local scopes in exec. Consider the following code, which raises NameError if the '#' is removed from the second last line. block = b = 'ok' def f(): print(b)# raises NameError here f() scope = locals()#.copy() exec(block, globals(),

[Python-Dev] autoconf update to 2.65 and cleanup

2010-04-23 Thread Matthias Klose
configure is still generated by 2.61; would it be possible to update to 2.65? The cr_lf issue mentioned in [1] seems to be resolved, ac_cr is now defined as ac_cr=`echo X | tr X '\015'` Proposing to - fix some quoting in help strings and code snippets (#8509) - update to autoconf 2.65

Re: [Python-Dev] autoconf update to 2.65 and cleanup

2010-04-23 Thread Victor Stinner
Le vendredi 23 avril 2010 17:26:59, Matthias Klose a écrit : configure is still generated by 2.61; would it be possible to update to 2.65? Yes, everything is possible. Open a new issue and write a patch ;-) even if 2.7 already is in beta? I'm not sure that it's a good idea to change the

Re: [Python-Dev] code.python.org - random 403 errors

2010-04-23 Thread Victor Stinner
Le jeudi 22 avril 2010 22:14:48, Sridhar Ratnakumar a écrit : I am seeing random 403 errors when cloning the mercurial repositories of Python. I don't know if it is related, but I get errors from the bbreport tool: -- $ python2.6 bbreport.py 3.x Selected builders: 20 / 80 (branch:

[Python-Dev] Summary of Python tracker Issues

2010-04-23 Thread Python tracker
ACTIVITY SUMMARY (2010-04-16 - 2010-04-23) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2665 open (+58) / 17664 closed (+31) / 20329 total (+89) Open issues with patches: 1084

Re: [Python-Dev] autoconf update to 2.65 and cleanup

2010-04-23 Thread Barry Warsaw
On Apr 23, 2010, at 05:44 PM, Victor Stinner wrote: I'm not sure that it's a good idea to change the build process after the first beta. It would depend on the issue comments ;-) OTOH, this doesn't seem like a new feature, so I think it should be okay. Doubly so if it fixes a bug. wink -Barry

Re: [Python-Dev] Inconsistent nesting of scopes in exec(..., locals())

2010-04-23 Thread Nick Coghlan
Joachim B Haga wrote: There seem to be an inconsistency in the handling of local scopes in exec. Consider the following code, which raises NameError if the '#' is removed from the second last line. block = b = 'ok' def f(): print(b)# raises NameError here f() scope =

Re: [Python-Dev] code.python.org - random 403 errors

2010-04-23 Thread Sridhar Ratnakumar
On 2010-04-22, at 10:55 PM, Jeroen Ruigrok van der Werven wrote: -On [20100423 02:48], Sridhar Ratnakumar (sridh...@activestate.com) wrote: Ok, I setup a cron job to maintain an internal mirror of the above mentioned repositories in code.python.org. We'll do a hg pull -u (equivalent to svn

[Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Dan Gindikin
We were having performance problems unpickling a large pickle file, we were getting 170s running time (which was fine), but 1100mb memory usage. Memory usage ought to have been about 300mb, this was happening because of memory fragmentation, due to many unnecessary puts in the pickle stream. We

Re: [Python-Dev] autoconf update to 2.65 and cleanup

2010-04-23 Thread Brett Cannon
On Fri, Apr 23, 2010 at 09:21, Barry Warsaw ba...@python.org wrote: On Apr 23, 2010, at 05:44 PM, Victor Stinner wrote: I'm not sure that it's a good idea to change the build process after the first beta. It would depend on the issue comments ;-) OTOH, this doesn't seem like a new feature,

Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Brett Cannon
On Fri, Apr 23, 2010 at 11:11, Dan Gindikin dgindi...@gmail.com wrote: We were having performance problems unpickling a large pickle file, we were getting 170s running time (which was fine), but 1100mb memory usage. Memory usage ought to have been about 300mb, this was happening because of

Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Alexandre Vassalotti
On Fri, Apr 23, 2010 at 2:11 PM, Dan Gindikin dgindi...@gmail.com wrote: We were having performance problems unpickling a large pickle file, we were getting 170s running time (which was fine), but 1100mb memory usage. Memory usage ought to have been about 300mb, this was happening because of

Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Alexandre Vassalotti
On Fri, Apr 23, 2010 at 2:38 PM, Alexandre Vassalotti alexan...@peadrop.com wrote: Collin Winter wrote a simple optimization pass for cPickle in Unladen Swallow [1]. The code reads through the stream and remove all the unnecessary PUTs in-place. I just noticed the code removes *all* PUT

Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Dan Gindikin
Alexandre Vassalotti alexandre at peadrop.com writes: Just put your code on bugs.python.org and I will take a look. Thanks, I'll put it in there. ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Collin Winter
On Fri, Apr 23, 2010 at 11:49 AM, Alexandre Vassalotti alexan...@peadrop.com wrote: On Fri, Apr 23, 2010 at 2:38 PM, Alexandre Vassalotti alexan...@peadrop.com wrote: Collin Winter wrote a simple optimization pass for cPickle in Unladen Swallow [1]. The code reads through the stream and remove

Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Collin Winter
On Fri, Apr 23, 2010 at 11:53 AM, Collin Winter collinwin...@google.com wrote: On Fri, Apr 23, 2010 at 11:49 AM, Alexandre Vassalotti alexan...@peadrop.com wrote: On Fri, Apr 23, 2010 at 2:38 PM, Alexandre Vassalotti alexan...@peadrop.com wrote: Collin Winter wrote a simple optimization pass

Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Alexandre Vassalotti
On Fri, Apr 23, 2010 at 3:07 PM, Collin Winter collinwin...@google.com wrote: I should add that, adding the necessary bookkeeping to remove only unused PUTs (instead of the current all-or-nothing scheme) should not be hard. I'd watch out for a further performance/memory hit; the pickling

Re: [Python-Dev] Inconsistent nesting of scopes in exec(..., locals())

2010-04-23 Thread Joachim B Haga
Nick Coghlan ncogh...@gmail.com writes: Joachim B Haga wrote: There seem to be an inconsistency in the handling of local scopes in exec. [...] The intermediate scope is searched for the variable name if the third argument to exec() is locals(), but not if it is locals().copy(). What

Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Dan Gindikin
Collin Winter collinwinter at google.com writes: I should add that, adding the necessary bookkeeping to remove only unused PUTs (instead of the current all-or-nothing scheme) should not be hard. I'd watch out for a further performance/memory hit; the pickling benchmarks in the benchmark suite

Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Alexandre Vassalotti
On Fri, Apr 23, 2010 at 3:57 PM, Dan Gindikin dgindi...@gmail.com wrote: This wouldn't help our use case, your code needs the entire pickle stream to be in memory, which in our case would be about 475mb, this is on top of the 300mb+ data structures that generated the pickle stream. In that

Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Dan Gindikin
Alexandre Vassalotti alexandre at peadrop.com writes: On Fri, Apr 23, 2010 at 3:57 PM, Dan Gindikin dgindikin at gmail.com wrote: This wouldn't help our use case, your code needs the entire pickle stream to be in memory, which in our case would be about 475mb, this is on top of the

Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Collin Winter
On Fri, Apr 23, 2010 at 1:53 PM, Alexandre Vassalotti alexan...@peadrop.com wrote: On Fri, Apr 23, 2010 at 3:57 PM, Dan Gindikin dgindi...@gmail.com wrote: This wouldn't help our use case, your code needs the entire pickle stream to be in memory, which in our case would be about 475mb, this is

Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Antoine Pitrou
Collin Winter collinwinter at google.com writes: I don't think it's possible in general to remove any PUTs if the pickle is being written to a file-like object. Does cPickle bytecode have some kind of NOP instruction? You could keep track of which PUTs weren't necessary and zero them out at

Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Dan Gindikin
Collin Winter collinwinter at google.com writes: I don't think it's possible in general to remove any PUTs if the pickle is being written to a file-like object. It is possible to reuse a single Pickler to pickle multiple objects: this causes the Pickler's memo dict to be shared between the

Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Dan Gindikin
Antoine Pitrou solipsis at pitrou.net writes: Does cPickle bytecode have some kind of NOP instruction? You could keep track of which PUTs weren't necessary and zero them out at the end. It would be much cheaper than writing a whole other optimized stream. For a large file, I'm not sure it is

Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution

2010-04-23 Thread Antoine Pitrou
Dan Gindikin dgindikin at gmail.com writes: Antoine Pitrou solipsis at pitrou.net writes: Does cPickle bytecode have some kind of NOP instruction? You could keep track of which PUTs weren't necessary and zero them out at the end. It would be much cheaper than writing a whole other