Re: Most efficient way to pre-grow a list?

2009-11-11 Thread Francis Carr
Hmmm. I am trying some algorithms, and timeit reports that a list.extend variant is fastest... WTH?! Really this seems like it must be a bug in implementing the [None]*x idiom. As expected, appending one-at-a-time is slowest (by an order of magnitude): % python -m timeit -s N=100 \

Re: Most efficient way to pre-grow a list?

2009-11-11 Thread Steven D'Aprano
On Wed, 11 Nov 2009 08:13:54 -0800, Francis Carr wrote: Hmmm. I am trying some algorithms, and timeit reports that a list.extend variant is fastest... WTH?! Really this seems like it must be a bug in implementing the [None]*x idiom. The straightforward pythonic solution is better yet: %

Re: Most efficient way to pre-grow a list?

2009-11-10 Thread gil_johnson
On Nov 9, 10:56 pm, Gabriel Genellina wrote: [much cutting] def method3a():    newArray = array.array('I', [INITIAL_VALUE]) * SIZE    assert len(newArray)==SIZE    assert newArray[SIZE-1]==INITIAL_VALUE [more cutting] So arrays are faster than lists, and in both cases one_item*N

Re: Most efficient way to pre-grow a list?

2009-11-09 Thread Gabriel Genellina
En Sun, 08 Nov 2009 10:08:35 -0300, gil_johnson gil_john...@earthlink.net escribió: On Nov 6, 8:46 pm, gil_johnson gil_john...@earthlink.net wrote: The problem I was solving was this: I wanted an array of 32-bit integers to be used as a bit array, and I wanted it initialized with all bits set,

Re: Most efficient way to pre-grow a list?

2009-11-08 Thread gil_johnson
On Nov 6, 8:46 pm, gil_johnson gil_john...@earthlink.net wrote: I don't have the code with me, but for huge arrays, I have used something like: arr[0] = initializer for i in range N:      arr.extend(arr) This doubles the array every time through the loop, and you can add the powers of

Re: Most efficient way to pre-grow a list?

2009-11-08 Thread gb345
In mailman.44.1257626420.2873.python-l...@python.org Luis Alberto Zarrabeitia Gomez ky...@uh.cu writes: ¿Have you ever tried to read list/matrix that you know it is not sparse, but you jdon't know the size, and it may not be in order? A grow-able array would just be the right thing to use -

Re: Most efficient way to pre-grow a list?

2009-11-08 Thread Emile van Sebille
On 11/7/2009 5:18 PM Carl Banks said... I think the top one is O(N log N), and I'm suspicious that it's even possible to grow a list in less than O(N log N) time without knowing the final size in advance. Citation? Quoting Tim -- Python uses mildly exponential over-allocation, sufficient

Re: Most efficient way to pre-grow a list?

2009-11-07 Thread Steven D'Aprano
On Fri, 06 Nov 2009 18:46:33 -0800, gil_johnson wrote: I don't have the code with me, but for huge arrays, I have used something like: arr[0] = initializer for i in range N: arr.extend(arr) This doubles the array every time through the loop, and you can add the powers of 2 to get

Re: Most efficient way to pre-grow a list?

2009-11-07 Thread Bruno Desthuilliers
kj a écrit : As I said, this is considered an optimization, at least in Perl, because it lets the interpreter allocate all the required memory in one fell swoop, instead of having to reallocate it repeatedly as the array grows. IIRC, CPython has it's own way to optimize list growth. (Of

Re: Most efficient way to pre-grow a list?

2009-11-07 Thread Luis Alberto Zarrabeitia Gomez
Quoting Bruno Desthuilliers bdesth.quelquech...@free.quelquepart.fr: Another situation where one may want to do this is if one needs to initialize a non-sparse array in a non-sequential order, Then use a dict. Ok, he has a dict. Now what? He needs a non-sparse array. -- Luis

Re: Most efficient way to pre-grow a list?

2009-11-07 Thread Andre Engels
On Sat, Nov 7, 2009 at 8:25 PM, Luis Alberto Zarrabeitia Gomez ky...@uh.cu wrote: Quoting Bruno Desthuilliers bdesth.quelquech...@free.quelquepart.fr: Another situation where one may want to do this is if one needs to initialize a non-sparse array in a non-sequential order, Then use a

Re: Most efficient way to pre-grow a list?

2009-11-07 Thread Terry Reedy
Steven D'Aprano wrote: On Fri, 06 Nov 2009 18:46:33 -0800, gil_johnson wrote: I don't have the code with me, but for huge arrays, I have used something like: arr[0] = initializer for i in range N: arr.extend(arr) This doubles the array every time through the loop, and you can add the

Re: Most efficient way to pre-grow a list?

2009-11-07 Thread Ivan Illarionov
On Nov 6, 3:12 pm, kj no.em...@please.post wrote: The best I can come up with is this: arr = [None] * 100 Is this the most efficient way to achieve this result? It is the most efficient SAFE way to achieve this result. In fact, there IS the more efficient way, but it's dangerous,

Re: Most efficient way to pre-grow a list?

2009-11-07 Thread Carl Banks
On Nov 7, 5:05 pm, sturlamolden sturlamol...@yahoo.no wrote: On 7 Nov, 03:46, gil_johnson gil_john...@earthlink.net wrote: I don't have the code with me, but for huge arrays, I have used something like: arr[0] = initializer for i in range N:      arr.extend(arr) This doubles the

Re: Most efficient way to pre-grow a list?

2009-11-07 Thread exarkun
On 01:18 am, pavlovevide...@gmail.com wrote: On Nov 7, 5:05�pm, sturlamolden sturlamol...@yahoo.no wrote: On 7 Nov, 03:46, gil_johnson gil_john...@earthlink.net wrote: I don't have the code with me, but for huge arrays, I have used something like: arr[0] = initializer for i in range N:

Re: Most efficient way to pre-grow a list?

2009-11-07 Thread Paul Rudin
kj no.em...@please.post writes: The best I can come up with is this: arr = [None] * 100 Is this the most efficient way to achieve this result? Depends on what you take as given. You can do it with ctypes more efficiently, but you can also shoot yourself in the foot. Another