Re: Optimizing list processing

2013-12-12 Thread rusi
On Friday, December 13, 2013 8:31:37 AM UTC+5:30, Steven D'Aprano wrote: > I don't know of any reasonable way to tell at runtime which of the two > algorithms I ought to take. Hard-coding an arbitrary value > ("if len(table) > 500") is not the worst idea I've ever had, but I'm > hoping for

Re: Optimizing list processing

2013-12-12 Thread Steven D'Aprano
On Thu, 12 Dec 2013 16:08:33 +0100, Peter Otten wrote: > Steven D'Aprano wrote: [...] >> So, ideally I'd like to write my code like this: >> >> >> table = [(x, i) for i,x in enumerate(iterable)] table.sort() >> if len(table) < ?: >> table = [i for x,i in table] >> else: >> for x, i i

Re: Optimizing list processing

2013-12-12 Thread Chris Angelico
On Fri, Dec 13, 2013 at 11:14 AM, Steven D'Aprano wrote: > Back in the 1980s, when I was a Mac user and occasional programmer, there > were memory manager routines which (if I recall correctly) would tell you > whether or not an allocation would succeed or not. Macs of that vintage > didn't have v

Re: Optimizing list processing

2013-12-12 Thread Steven D'Aprano
t functionality which was standard in 1984 computers is not possible in 2013. But I don't *specifically* care about measuring memory size, only as a proxy for whether or not I should switch algorithms. That's why this post is called "Optimizing list processing" rather than "F

Re: Optimizing list processing

2013-12-12 Thread Terry Reedy
On 12/12/2013 7:08 AM, Steven D'Aprano wrote: Please don't focus on the algorithm I gave. Focus on the fact that I could write it like this: if some condition to do with the computer's available memory: make modifications in place else: make a copy of the data containing the modific

Re: Optimizing list processing

2013-12-12 Thread Terry Reedy
On 12/12/2013 6:09 AM, Stefan Behnel wrote: Terry Reedy, 12.12.2013 03:26: from itertools import count table = sorted(t for t in zip(iterable, count)) This is a useless use of a generator expression. sorted(zip(...)) is enough (and likely to be substantially faster also). Yes, definitely, an

Re: Optimizing list processing

2013-12-12 Thread Peter Otten
Steven D'Aprano wrote: > I have some code which produces a list from an iterable using at least > one temporary list, using a Decorate-Sort-Undecorate idiom. The algorithm > looks something like this (simplified): > > table = sorted([(x, i) for i,x in enumerate(iterable)]) > table = [i for x,i in

Re: Optimizing list processing

2013-12-12 Thread Chris Angelico
On Fri, Dec 13, 2013 at 12:32 AM, MRAB wrote: >> table[:] = [i for x,i in table] # Does slice assignment get optimized? >> > [snip] > > If you're trying that, you could also try: > > table[:] = (i for x,i in table) Right, a tweak which could be applied also to the split version. Thanks MRAB. Ch

Re: Optimizing list processing

2013-12-12 Thread MRAB
On 12/12/2013 12:25, Chris Angelico wrote: On Thu, Dec 12, 2013 at 11:08 PM, Steven D'Aprano wrote: P.S. The algorithm I'm working on is a way of generating index and rank tables. Not that it really matters -- what matters is determining whether or not to shift from "make a copy of the list" to

Re: Optimizing list processing

2013-12-12 Thread Chris Angelico
On Thu, Dec 12, 2013 at 11:08 PM, Steven D'Aprano wrote: > P.S. The algorithm I'm working on is a way of generating index and rank > tables. Not that it really matters -- what matters is determining whether > or not to shift from "make a copy of the list" to "modify the list in > place". So you'r

Re: Optimizing list processing

2013-12-12 Thread Steven D'Aprano
On Wed, 11 Dec 2013 21:26:51 -0500, Terry Reedy wrote: > On 12/11/2013 6:54 PM, Steven D'Aprano wrote: >> I have some code which produces a list from an iterable using at least >> one temporary list, using a Decorate-Sort-Undecorate idiom. > > It is a non-standard version thereof, as DSU usually

Re: Optimizing list processing

2013-12-12 Thread Stefan Behnel
Terry Reedy, 12.12.2013 03:26: > from itertools import count > table = sorted(t for t in zip(iterable, count)) This is a useless use of a generator expression. sorted(zip(...)) is enough (and likely to be substantially faster also). Stefan -- https://mail.python.org/mailman/listinfo/python-lis

Re: Optimizing list processing

2013-12-11 Thread Terry Reedy
On 12/11/2013 6:54 PM, Steven D'Aprano wrote: I have some code which produces a list from an iterable using at least one temporary list, using a Decorate-Sort-Undecorate idiom. It is a non-standard version thereof, as DSU usually means to decorate with a key that gets discarded. A couple of

Re: Optimizing list processing

2013-12-11 Thread MRAB
On 12/12/2013 01:43, Steven D'Aprano wrote: On Thu, 12 Dec 2013 00:59:42 +, MRAB wrote: table = [(x, i) for i,x in enumerate(iterable)] table.sort() This looks wrong to me: for x, i in table: table[i] = x Yes, you're right, I over-simplified the example, and in doing so introduce

Re: Optimizing list processing

2013-12-11 Thread Steven D'Aprano
On Thu, 12 Dec 2013 00:59:42 +, MRAB wrote: >> table = [(x, i) for i,x in enumerate(iterable)] >> table.sort() > > This looks wrong to me: > >> for x, i in table: >> table[i] = x Yes, you're right, I over-simplified the example, and in doing so introduced a bug. What I actually use i

Re: Optimizing list processing

2013-12-11 Thread Ben Finney
Steven D'Aprano writes: > For giant iterables (ten million items), this version is a big > improvement, about three times faster than the list comp version. […] > > Except that for more reasonably sized iterables, it's a pessimization. > With one million items, the ratio is the other way around:

Re: Optimizing list processing

2013-12-11 Thread duncan smith
On 11/12/13 23:54, Steven D'Aprano wrote: I have some code which produces a list from an iterable using at least one temporary list, using a Decorate-Sort-Undecorate idiom. The algorithm looks something like this (simplified): table = sorted([(x, i) for i,x in enumerate(iterable)]) table = [i fo

Re: Optimizing list processing

2013-12-11 Thread MRAB
On 11/12/2013 23:54, Steven D'Aprano wrote: I have some code which produces a list from an iterable using at least one temporary list, using a Decorate-Sort-Undecorate idiom. The algorithm looks something like this (simplified): table = sorted([(x, i) for i,x in enumerate(iterable)]) table = [i

Optimizing list processing

2013-12-11 Thread Steven D'Aprano
I have some code which produces a list from an iterable using at least one temporary list, using a Decorate-Sort-Undecorate idiom. The algorithm looks something like this (simplified): table = sorted([(x, i) for i,x in enumerate(iterable)]) table = [i for x,i in table] The problem here is that