[Tutor] list comprehension efficiency
Generally speaking, are list comprehensions more efficient that the equivalent for loop with interior conditionals for a given task? Do they compile down to the same Python byte code? Thanks, Bill Allen ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension efficiency
On 19/02/2012 01:31, Bill Allen wrote: Generally speaking, are list comprehensions more efficient that the equivalent for loop with interior conditionals for a given task? Do they compile down to the same Python byte code? Thanks, Bill Allen ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor Use the timeit module to answer Q1 and the dis module to answer Q2. -- Cheers. Mark Lawrence. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension efficiency
Bill Allen wrote: Generally speaking, are list comprehensions more efficient that the equivalent for loop with interior conditionals for a given task? Do they compile down to the same Python byte code? It depends, and no. For-loops are more general, so there are loops that can't be written as list comprehensions. Consequently, so is the byte code generated. Python makes no attempt to analyse a for-loop and decide that it could be written as a list comp. Nor do list comps generate the exact same code as a for-loop: they can't, because the semantics are slightly different. (Specifically, a list comp is an atomic operation: either the whole thing succeeds, or the resultant list does not persist. That same does not apply to for-loops.) You can inspect the byte code produced by using the dis module. But, for the subset of possible for-loops which can be re-written as list comps, yes, list comps often are more efficient, as building the list can be performed at full C speed instead of at relatively slower Python speed. But don't over-estimate how big a difference that makes in practice. For trivial loops, it makes a big difference: py with Timer(): ... _ = [None for i in range(1)] ... time taken: 0.004692 seconds py py with Timer(): ... _ = [] ... for i in range(1): ... _.append(None) ... time taken: 0.020877 seconds but for loops that do actual, significant work, the difference becomes correspondingly less important: py def func(x): ... y = 2*i**3 - 15*i*i + 17*i + 3 ... z = x**0.5/(2*y**2 - 1) ... return (3*z)/(1+z)**1.25 ... py with Timer(): ... _ = [func(i) for i in range(1)] ... time taken: 0.210438 seconds py py with Timer(): ... _ = [] ... for i in range(1): ... _.append(func(i)) ... time taken: 0.250344 seconds If you're interested in using my Timer() code, you can find it here: http://code.activestate.com/recipes/577896-benchmark-code-with-the-with-statement/ Beware though, that you can't exit out of a list comp early. So if you have code like this: collection = [] for x in sorted(sequence): if x 10: break collection.append(x + 5) versus this: collection = [x+5 for x in sorted(sequence) if x = 10] Consider what happens if sequence = range(100). The for-loop does eleven iterations, the list-comp does a million. Guess which will be faster? :) -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, efficiency?
On 10/2/2010 8:02 PM, Steven D'Aprano wrote: On Sun, 3 Oct 2010 01:17:39 am bob gailer wrote: I ran dis on a for loop and the equivalent comprehension. I was surprised to see almost identical code. I had assumed (and now wish for) that a comprehension would be a primitive written in C and thus much faster! How could it be? A list comp is syntactic sugar. It needs to perform the same steps as a for-loop, and call an arbitrary Python expression. It's not like map() that takes a function object. Unless you had a separate primitive for every imaginable Python expression -- which is impossible -- list comps need to generate relatively similar code to for-loops because they do relatively similar things. Thank you. I needed that! Besides, in recent versions of Python the speed of for-loops is quite good. The bottleneck is rarely the loop itself, but the work you do in the loop or the size of the loop. -- Bob Gailer 919-636-4239 Chapel Hill NC ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, efficiency?
[snip] I ran dis on a for loop and the equivalent comprehension. I was surprised to see almost identical code. I had assumed (and now wish for) that a comprehension would be a primitive written in C and thus much faster! -- Bob Gailer 919-636-4239 Chapel Hill NC ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, efficiency?
On Sun, 3 Oct 2010 01:17:39 am bob gailer wrote: I ran dis on a for loop and the equivalent comprehension. I was surprised to see almost identical code. I had assumed (and now wish for) that a comprehension would be a primitive written in C and thus much faster! How could it be? A list comp is syntactic sugar. It needs to perform the same steps as a for-loop, and call an arbitrary Python expression. It's not like map() that takes a function object. Unless you had a separate primitive for every imaginable Python expression -- which is impossible -- list comps need to generate relatively similar code to for-loops because they do relatively similar things. Besides, in recent versions of Python the speed of for-loops is quite good. The bottleneck is rarely the loop itself, but the work you do in the loop or the size of the loop. -- Steven D'Aprano ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, efficiency?
On 09/28/10 13:57, Bill Allen wrote: I can now see that quite a bit of the code I write dealing with lists can be done with list comprehensions. My question is this, is the list comprehension styled code generally more efficient at runtime? If so, why? Yes, because the looping in list comprehension is done in C instead of a python construct. However, they are the type of efficiency that you shouldn't bother yourself with unless you're microoptimizing. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, efficiency?
On Tue, 28 Sep 2010 01:57:23 pm Bill Allen wrote: I can now see that quite a bit of the code I write dealing with lists can be done with list comprehensions. My question is this, is the list comprehension styled code generally more efficient at runtime? If so, why? List comprehensions *are* lists, or rather, they produce lists. A list comprehension result = [expr for x in seq] is just syntactic sugar for a for-loop: result = [] for x in seq: result.append(x) except that in Python 3, the x variable is hidden. (In Python 2 it is not, but that was an accident.) The end result is exactly the same whether you use a list comp or a for-loop. The advantage of for-loops is that you can do much more complex code. That complexity comes at a cost, and consequently for-loops tend to be a little bit slower than list comprehensions: some of the work can be done under the hood, faster than regular Python code. But for most cases, this difference is relatively small and won't make any real difference. What does it matter if your program runs in 24 milliseconds instead of 22 milliseconds? Or five hours and seventeen minutes instead of five hours and sixteen minutes? Who cares? Write the code that is most natural and easy to read and understand, and only worry about such tiny savings when you absolutely have to. But there is one case where for-loops are potentially MUCH faster than list comps. List comps always run all the way to the end, but for-loops can break out early. If your problem lets you break out of the loop early, this is a good thing. So this for-loop: for x in xrange(1000): result.append(x+1) if x 5: break will beat the pants off this list comp: [x+1 for x in xrange(1000) if x = 5] There's no way to break out of the list comp early -- it has to keep going past 5 all the way to the end. -- Steven D'Aprano ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] list comprehension, efficiency?
On Tue, Sep 28, 2010, Lie Ryan wrote: On 09/28/10 13:57, Bill Allen wrote: I can now see that quite a bit of the code I write dealing with lists can be done with list comprehensions. My question is this, is the list comprehension styled code generally more efficient at runtime? If so, why? Yes, because the looping in list comprehension is done in C instead of a python construct. However, they are the type of efficiency that you shouldn't bother yourself with unless you're microoptimizing. True enough, but dealing with large lists can be expensive, particularly when appending. Before I discovered sets, I found significant time savings by creating a dictionary, adding as necessary, then use d.keys() to get the keys back. This became a significant factor when I was selecting unique lines from about 1.3 million samples. Bill -- INTERNET: b...@celestial.com Bill Campbell; Celestial Software LLC URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way Voice: (206) 236-1676 Mercer Island, WA 98040-0820 Fax:(206) 232-9186 Skype: jwccsllc (206) 855-5792 Independent self-reliant people would be a counterproductive anachronism in the collective society of the future where people will be defined by their associations. 1896 John Dewey, educational philosopher, proponent of modern public schools. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] list comprehension, efficiency?
I have seen list comprehensions used, but have not quite got the hang of it yet. So, I was writing a bit of code to do some work with file directories and decided to give it a try as follows: list_c = os.listdir(c:) #first code written in the way I usually would. dirs = [] for x in list_c: if os.path.isdir(x): dirs.append(x) #replaced the above code with the following list comprehension, it worked as expected: dirs = [x for x in list_c if os.path.isdir(x)] I can now see that quite a bit of the code I write dealing with lists can be done with list comprehensions. My question is this, is the list comprehension styled code generally more efficient at runtime? If so, why? --Bill ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
