Re: [Python-Dev] Proposal: Run GC less often

2008-06-21 Thread Martin v. Löwis
> What happens if the program enters a phase where it's not > producing any new cyclic garbage, but is breaking references > among the old objects in such a way that cycles of them > are being left behind? Under this rule, the oldest > generation would never be scanned, so those cycles would > neve

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Stephen J. Turnbull
"Martin v. Löwis" writes: > > XEmacs implements this strategy in a way which is claimed to give > > constant amortized time (ie, averaged over memory allocated). > > See my recent proposal. I did, crossed in the mail. To the extent that I understand both systems, your proposal looks like an

Re: [Python-Dev] Another Proposal: Run GC less often

2008-06-21 Thread Tony Nelson
At 11:28 PM +0200 6/21/08, none wrote: >Instead of collecting objects after a fixed number of allocations (700) ... I've seen this asserted several times in this thread: that GC is done every fixed number of allocations. This is not correct. GC is done when the surplus of allocations less deal

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Nick Coghlan
Martin v. Löwis wrote: Antoine Pitrou wrote: Le samedi 21 juin 2008 à 17:49 +0200, "Martin v. Löwis" a écrit : I don't think any strategies based on timing will be successful. Instead, one should count and analyze objects (although I'm unsure how exactly that could work). Would it be helpful i

Re: [Python-Dev] Proposal: Run GC less often

2008-06-21 Thread Greg Ewing
Martin v. Löwis wrote: Under my proposal, 10 middle collections must have passed, PLUS the number of survivor objects from the middle generation must exceed 10% of the number of objects in the oldest generation. What happens if the program enters a phase where it's not producing any new cyclic

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Martin v. Löwis
> XEmacs implements this strategy in a way which is claimed to give > constant amortized time (ie, averaged over memory allocated). See my recent proposal. The old trick is to do reorganizations in a fixed fraction of the total size, resulting in a per-increase amortized-constant overhead (assumin

Re: [Python-Dev] [Python-3000] RELEASED Python 2.6b1 and 3.0b1

2008-06-21 Thread Barry Warsaw
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Jun 21, 2008, at 5:45 PM, Leif Walsh wrote: There is a typo on the download page (http://python.org/download/releases/3.0/ ): Fixed, thanks! - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSF2G5nEjvBPtnXfVAQJbqwQ

Re: [Python-Dev] [Python-3000] RELEASED Python 2.6b1 and 3.0b1

2008-06-21 Thread Leif Walsh
There is a typo on the download page (http://python.org/download/releases/3.0/): """The release plan is to have a series of alpha releases in 2007, beta releases in 2008, and a final release in September 2008. The alpha releases are primarily aimed at developers who want a sneak peek at the new la

Re: [Python-Dev] [Python-3000] RELEASED Python 2.6b1 and 3.0b1

2008-06-21 Thread Leif Walsh
There is a typo on the download page (http://python.org/download/releases/3.0/): """The release plan is to have a series of alpha releases in 2007, beta releases in 2008, and a final release in September 2008. The alpha releases are primarily aimed at developers who want a sneak peek at the new la

Re: [Python-Dev] Proposal: Run GC less often

2008-06-21 Thread Leif Walsh
If you can get me a version of the interpreter with this change made (I wouldn't know what to change), I can run a very allocation/deallocation-heavy application I have lying around, and get you some good benchmarks. On Sat, Jun 21, 2008 at 1:23 PM, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: - S

[Python-Dev] Another Proposal: Run GC less often

2008-06-21 Thread none
Instead of collecting objects after a fixed number of allocations (700) You could make it dynamic like this: # initial values min_allocated_memory = 0 max_allocated_memory = 0 next_gc_run = 1024 * 1024 def manage_memory(): allocated_memory = get_allocated_memory() min_allocated_memory =

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Stephen J. Turnbull
"Martin v. Löwis" writes: > Given the choice of "run slower" and "run out of memory", Python should > always prefer the former. > > One approach could be to measure how successful a GC run was: if GC > finds that more-and-more objects get allocated and very few (or none) > are garbage, it m

Re: [Python-Dev] Proposal: Run GC less often

2008-06-21 Thread Brett Cannon
On Sat, Jun 21, 2008 at 1:23 PM, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > Here is my proposal for making the GC run less often. > The objective is to avoid the quadratic-time behavior > if many objects are created and none of them is garbage. > [SNIP] > Applications with a small number of obj

[Python-Dev] Proposal: Run GC less often

2008-06-21 Thread Martin v. Löwis
Here is my proposal for making the GC run less often. The objective is to avoid the quadratic-time behavior if many objects are created and none of them is garbage. The youngest generation remains collected after 700 allocations, the middle generation after 10 young collections. Therefore, full GC

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Martin v. Löwis
Antoine Pitrou wrote: > Le samedi 21 juin 2008 à 17:49 +0200, "Martin v. Löwis" a écrit : >> I don't think any strategies based on timing will be successful. >> Instead, one should count and analyze objects (although I'm unsure >> how exactly that could work). > > Would it be helpful if the GC wa

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Antoine Pitrou
Le samedi 21 juin 2008 à 17:49 +0200, "Martin v. Löwis" a écrit : > I don't think any strategies based on timing will be successful. > Instead, one should count and analyze objects (although I'm unsure > how exactly that could work). Would it be helpful if the GC was informed of memory growth by

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Terry Reedy
Kevin Jacobs <[EMAIL PROTECTED]> wrote: I can say with complete certainty that of the 20+ programmers I've had working for me, many who have used Python for 3+ years, not a single one would think to question the garbage collector if they observed the kind of quadratic time complexity I've de

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Bill Janssen
> > What follows from that? To me, the natural conclusion is "people who > > witness performance problems just need to despair, or accept them, as > > they can't do anything about it", however, I don't think this is the > > conclusion that you had in mind. > > > > I can say with complete certainty

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Martin v. Löwis
> I'm not sure I agree with this. GC IIRC was introduced primarily to > alleviate *long-term* memory starvation. I don't think that's historically the case. GC would not need to be generational if releasing short-lived objects shortly after they become garbage was irrelevant. Of course, it was al

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Aahz
On Sat, Jun 21, 2008, "Martin v. L??wis" wrote: > > In general, any solution of the "do GC less often" needs to deal with > cases where lots of garbage gets produced in a short amount of time > (e.g. in a tight loop), and which run out of memory when GC is done less > often. > > Given the choice

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Martin v. Löwis
> Idea 1: Allow GC to run automatically no more often than n CPU seconds, > n being perhaps 5 or 10. I think it's very easy to exhaust the memory with such a policy, even though much memory would still be available. Worse, in a program producing a lot of garbage, performance will go significantly

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Kevin Jacobs <[EMAIL PROTECTED]>
On Sat, Jun 21, 2008 at 11:20 AM, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > In general, any solution of the "do GC less often" needs to deal with > cases where lots of garbage gets produced in a short amount of time > (e.g. in a tight loop), and which run out of memory when GC is done less >

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Martin v. Löwis
> Well, they could hang themselves or switch to another language (which > some people might view as equivalent :-)), but perhaps optimistically > the various propositions that were sketched out in this thread (by Adam > Olsen and Greg Ewing) could bring an improvement. I don't know how > realistic

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Martin v. Löwis
> I can say with complete certainty that of the 20+ programmers I've had > working for me, many who have used Python for 3+ years, not a single one > would think to question the garbage collector if they observed the kind > of quadratic time complexity I've demonstrated. This is not because > they

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Kevin Jacobs <[EMAIL PROTECTED]>
On Sat, Jun 21, 2008 at 4:33 AM, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > I don't think expecting people to tweak gc parameters when they witness > > performance problems is reasonable. > > What follows from that? To me, the natural conclusion is "people who > witness performance problems

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Antoine Pitrou
Le samedi 21 juin 2008 à 10:33 +0200, "Martin v. Löwis" a écrit : > > I don't think expecting people to tweak gc parameters when they witness > > performance problems is reasonable. > > What follows from that? To me, the natural conclusion is "people who > witness performance problems just need to

[Python-Dev] csv module and Unicode in 2.6 and 3.0

2008-06-21 Thread skip
If I understand things correctly, the csv module in 3.0 supports Unicode, kind of for free because all strings are Unicode. I suspect the Unicode examples at the bottom of the 3.0 version of the module doc need to be removed. Does the 2.6 version support Unicode out of the box as well or are the

[Python-Dev] Issue 2722

2008-06-21 Thread Neil Muller
Could some one have a look at the suggested fix for issue 2722? While not a particularly common problem, it can be an annoyance, and the patch looks reasonable. -- Neil Muller ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mail

Re: [Python-Dev] Status of Issue 2331 - Backport parameter annotations

2008-06-21 Thread [EMAIL PROTECTED]
> Does it bother you that you need ()s to make instances elsewhere? > That you have to type int('123') instead of int, '123'? Not at all...Python never supported this. Cheers, David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.or

Re: [Python-Dev] C API for gc.enable() and gc.disable()

2008-06-21 Thread Martin v. Löwis
> I don't think expecting people to tweak gc parameters when they witness > performance problems is reasonable. What follows from that? To me, the natural conclusion is "people who witness performance problems just need to despair, or accept them, as they can't do anything about it", however, I do