Re: Pythonic way to iterate through multidimensional space?
On 10/7/2014 1:01 PM, Ned Batchelder wrote: On 10/7/14 2:10 AM, Gelonida N wrote: Disadvantage of itertools.product() is, that it makes a copy in memory. Reason ist, that itertools also makes products of generators (meaning of objects, that one can't iterate several times through) There are two use cases, that I occasionaly stumble over: One is making the product over lists(), product( list_of_lists ) ex: product( [ [1,2,3], ['A','B'], ['a', 'b', 'c'] ] ) the other one making a product over a list of functions, which will create generators ex: product( [ lambda: [ 'A', 'B' ], lambda: xrange(3) ] ) I personally would also be interested in a fast generic solution that can iterate through an N-dimensional array and which does not duplicate the memory or iterate through a list of generator-factories or however this would be called. itertools.product makes a copy of the sequences passed in, but it is a shallow copy. It doesn't copy the objects in the sequences. It also doesn't store the entire product. If you are calling product(j, k, l, m, n), where len(j)==J, the extra memory is J+K+L+M+N, which is much smaller than the number of iterations product will produce. Are you sure that much extra memory use is a problem? How large are your lists that you are product'ing together? Thanks for the clarification. You are right. Consumption of a shallow copy of each iterator, should not be a problem in the cases that I encountered so far. I don't understand your point about a list of functions that create generators? What is the problem there? The idea was to even avoid the creation of a shallow copy, by having a function, that will return the same generator over and over again (thus no need for shallow copy) -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to iterate through multidimensional space?
On 8/6/2014 1:39 PM, Tim Chase wrote: On 2014-08-06 11:04, Gayathri J wrote: Below is the code I tried to check if itertools.product() was faster than normal nested loops... they arent! arent they supposed to be...or am i making a mistake? I believe something like this was discussed a while ago and there was a faster-but-uglier solution so you might want to consult this thread: https://mail.python.org/pipermail/python-list/2008-January/516109.html I believe this may have taken place before itertools.product() came into existence. Disadvantage of itertools.product() is, that it makes a copy in memory. Reason ist, that itertools also makes products of generators (meaning of objects, that one can't iterate several times through) There are two use cases, that I occasionaly stumble over: One is making the product over lists(), product( list_of_lists ) ex: product( [ [1,2,3], ['A','B'], ['a', 'b', 'c'] ] ) the other one making a product over a list of functions, which will create generators ex: product( [ lambda: [ 'A', 'B' ], lambda: xrange(3) ] ) I personally would also be interested in a fast generic solution that can iterate through an N-dimensional array and which does not duplicate the memory or iterate through a list of generator-factories or however this would be called. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to iterate through multidimensional space?
On 10/7/14 2:10 AM, Gelonida N wrote: Disadvantage of itertools.product() is, that it makes a copy in memory. Reason ist, that itertools also makes products of generators (meaning of objects, that one can't iterate several times through) There are two use cases, that I occasionaly stumble over: One is making the product over lists(), product( list_of_lists ) ex: product( [ [1,2,3], ['A','B'], ['a', 'b', 'c'] ] ) the other one making a product over a list of functions, which will create generators ex: product( [ lambda: [ 'A', 'B' ], lambda: xrange(3) ] ) I personally would also be interested in a fast generic solution that can iterate through an N-dimensional array and which does not duplicate the memory or iterate through a list of generator-factories or however this would be called. itertools.product makes a copy of the sequences passed in, but it is a shallow copy. It doesn't copy the objects in the sequences. It also doesn't store the entire product. If you are calling product(j, k, l, m, n), where len(j)==J, the extra memory is J+K+L+M+N, which is much smaller than the number of iterations product will produce. Are you sure that much extra memory use is a problem? How large are your lists that you are product'ing together? I don't understand your point about a list of functions that create generators? What is the problem there? -- Ned Batchelder, http://nedbatchelder.com -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to iterate through multidimensional space?
Dear Peter Below is the code I tried to check if itertools.product() was faster than normal nested loops... they arent! arent they supposed to be...or am i making a mistake? any idea? ** *# -*- coding: utf-8 -*-* *import numpy as np* *import time* *from itertools import product,repeat* *def main():* *# N - size of grid* *# nvel - number of velocities* *# times - number of times to run the functions* *N=256* *times=3* *f=np.random.rand(N,N,N)* *# using loops* *print normal nested loop* *python_dot_loop1(f,times,N)* *print nested loop using itertools.product()* *python_dot_loop2(f,times,N)* *def python_dot_loop1(f,times,N):* *for t in range(times):* * t1=time.time()* * for i in range(N):* * for j in range(N):* * for k in range(N):* * f[i,j,k] = 0.0* * print python dot loop + str(time.time()-t1)* *def python_dot_loop2(f,times,N):* *rangeN=range(N)* *for t in range(times):* * t1=time.time()* * for i,j,k in product(rangeN,repeat=3):* * f[i,j,k]=0.0* * print python dot loop + str(time.time()-t1)* *if __name__=='__main__':* *main()* ** -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to iterate through multidimensional space?
On Wed, Aug 6, 2014 at 3:34 PM, Gayathri J usethisid2...@gmail.com wrote: Below is the code I tried to check if itertools.product() was faster than normal nested loops... they arent! arent they supposed to be...or am i making a mistake? any idea? Don't worry about what's faster. That almost never matters. Worry, instead, about how you would code it if you can't be sure how many dimensions there'll be until run time (which the OP said can happen). With product(), you can give it a variable number of arguments (eg with *args notation), but with loops, you'd need to compile up some code with that many nested loops - or at best, something where you cap the number of loops and thus dimensions, and have a bunch of them iterate over a single iterable. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to iterate through multidimensional space?
On 06/08/2014 06:34, Gayathri J wrote: Dear Peter Below is the code I tried to check if itertools.product() was faster than normal nested loops... they arent! arent they supposed to be...or am i making a mistake? any idea? * * * * *# -*- coding: utf-8 -*- * *import numpy as np* *import time* *from itertools import product,repeat* *def main():* *# N - size of grid* *# nvel - number of velocities* *# times - number of times to run the functions* *N=256* *times=3* *f=np.random.rand(N,N,N)* ** *# using loops* *print normal nested loop* *python_dot_loop1(f,times,N)* * * *print nested loop using itertools.product()* *python_dot_loop2(f,times,N)* * * *def python_dot_loop1(f,times,N):* *for t in range(times):* * t1=time.time()* * for i in range(N):* * for j in range(N):* * for k in range(N):* * f[i,j,k] = 0.0* * print python dot loop + str(time.time()-t1)* ** *def python_dot_loop2(f,times,N):* *rangeN=range(N)* *for t in range(times):* * t1=time.time()* * for i,j,k in product(rangeN,repeat=3):* * f[i,j,k]=0.0* * print python dot loop + str(time.time()-t1)* * * * * *if __name__=='__main__':* *main()* ** Who cares, well not I for one? Give me slower but accurate code over faster but inaccurate code any day of the week? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to iterate through multidimensional space?
Gayathri J wrote: Dear Peter Below is the code I tried to check if itertools.product() was faster than normal nested loops... they arent! arent they supposed to be... I wouldn't have expected product() to be significantly faster, but neither did I expect it to be slower. or am i making a mistake? any idea? For your testcase you can shave off a small amount by avoiding the tuple- unpacking for t in product(...): f[t] = 0.0 but that may be cheating, and the effect isn't big. When you are working with numpy there may be specialised approaches, but f[:,:,:] = 0.0 is definitely cheating I suppose... -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to iterate through multidimensional space?
You might check numpy it is really powerful tool for working with multi dimensional arrays: ex. a = arange(81).reshape(3,3,3,3) a array( 0, 1, 2], [ 3, 4, 5], [ 6, 7, 8]], [[ 9, 10, 11], [12, 13, 14], [15, 16, 17]], [[18, 19, 20], [21, 22, 23], [24, 25, 26]]], [[[27, 28, 29], [30, 31, 32], [33, 34, 35]], [[36, 37, 38], [39, 40, 41], [42, 43, 44]], [[45, 46, 47], [48, 49, 50], [51, 52, 53]]], [[[54, 55, 56], [57, 58, 59], [60, 61, 62]], [[63, 64, 65], [66, 67, 68], [69, 70, 71]], [[72, 73, 74], [75, 76, 77], [78, 79, 80) f = a.flat for i in f: ...print(i) 0 1 2 .. 98 99 cheers Wojciech On 05/08/14 21:06, Frank Miles wrote: I need to evaluate a complicated function over a multidimensional space as part of an optimization problem. This is a somewhat general problem in which the number of dimensions and the function being evaluated can vary from problem to problem. I've got a working version (with loads of conditionals, and it only works to #dimensions = 10), but I'd like something simpler and clearer and less hard-coded. I've web-searched for some plausible method, but haven't found anything nice. Any recommendations where I should look, or what technique should be used? TIA! -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to iterate through multidimensional space?
Dear Peter Yes the f[t] or f[:,:,:] might give a marginal increase, but then i need to do further operations using the indices, in which case this wouldnt help Dear Wojciech np.flat() works if u dont care about the indices and only the matrix/array values matter. but if the i,j,k matters, flatten wouldnt work On Wed, Aug 6, 2014 at 1:34 PM, Wojciech Giel wojtekg...@gmail.com wrote: You might check numpy it is really powerful tool for working with multi dimensional arrays: ex. a = arange(81).reshape(3,3,3,3) a array( 0, 1, 2], [ 3, 4, 5], [ 6, 7, 8]], [[ 9, 10, 11], [12, 13, 14], [15, 16, 17]], [[18, 19, 20], [21, 22, 23], [24, 25, 26]]], [[[27, 28, 29], [30, 31, 32], [33, 34, 35]], [[36, 37, 38], [39, 40, 41], [42, 43, 44]], [[45, 46, 47], [48, 49, 50], [51, 52, 53]]], [[[54, 55, 56], [57, 58, 59], [60, 61, 62]], [[63, 64, 65], [66, 67, 68], [69, 70, 71]], [[72, 73, 74], [75, 76, 77], [78, 79, 80) f = a.flat for i in f: ...print(i) 0 1 2 .. 98 99 cheers Wojciech On 05/08/14 21:06, Frank Miles wrote: I need to evaluate a complicated function over a multidimensional space as part of an optimization problem. This is a somewhat general problem in which the number of dimensions and the function being evaluated can vary from problem to problem. I've got a working version (with loads of conditionals, and it only works to #dimensions = 10), but I'd like something simpler and clearer and less hard-coded. I've web-searched for some plausible method, but haven't found anything nice. Any recommendations where I should look, or what technique should be used? TIA! -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to iterate through multidimensional space?
Gayathri J wrote: Dear Peter Yes the f[t] or f[:,:,:] might give a marginal increase, The speedup compared itertools.product() is significant: $ python -m timeit -s 'from itertools import product; from numpy.random import rand; N = 100; a = rand(N, N, N); r = range(N)' 'for x in product(r, repeat=3): a[x] = 0.0' 10 loops, best of 3: 290 msec per loop $ python -m timeit -s 'from itertools import product; from numpy.random import rand; N = 100; a = rand(N, N, N); r = range(N)' 'a[:,:,:] = 0.0' 100 loops, best of 3: 3.58 msec per loop But normally you'd just make a new array with numpy.zeros(). but then i need to do further operations using the indices, in which case this wouldnt help Which is expected and also the crux of such micro-benchmarks. They distract from big picture. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to iterate through multidimensional space?
On 2014-08-06 11:04, Gayathri J wrote: Below is the code I tried to check if itertools.product() was faster than normal nested loops... they arent! arent they supposed to be...or am i making a mistake? I believe something like this was discussed a while ago and there was a faster-but-uglier solution so you might want to consult this thread: https://mail.python.org/pipermail/python-list/2008-January/516109.html I believe this may have taken place before itertools.product() came into existence. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to iterate through multidimensional space?
Dear Peter thanks . But thats what I was trying to say just taking them to zero by f[:,:,:] = 0.0 or using np.zeros is surely going to give me a time gain... but my example of using the itertools.product() and doing f[x] =0.0 is just to compare the looping timing with the traditional nested loops and not to distract us to the operation done inside the loop. right? On Wed, Aug 6, 2014 at 6:09 PM, Peter Otten __pete...@web.de wrote: Gayathri J wrote: Dear Peter Yes the f[t] or f[:,:,:] might give a marginal increase, The speedup compared itertools.product() is significant: $ python -m timeit -s 'from itertools import product; from numpy.random import rand; N = 100; a = rand(N, N, N); r = range(N)' 'for x in product(r, repeat=3): a[x] = 0.0' 10 loops, best of 3: 290 msec per loop $ python -m timeit -s 'from itertools import product; from numpy.random import rand; N = 100; a = rand(N, N, N); r = range(N)' 'a[:,:,:] = 0.0' 100 loops, best of 3: 3.58 msec per loop But normally you'd just make a new array with numpy.zeros(). but then i need to do further operations using the indices, in which case this wouldnt help Which is expected and also the crux of such micro-benchmarks. They distract from big picture. -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Pythonic way to iterate through multidimensional space?
I need to evaluate a complicated function over a multidimensional space as part of an optimization problem. This is a somewhat general problem in which the number of dimensions and the function being evaluated can vary from problem to problem. I've got a working version (with loads of conditionals, and it only works to #dimensions = 10), but I'd like something simpler and clearer and less hard-coded. I've web-searched for some plausible method, but haven't found anything nice. Any recommendations where I should look, or what technique should be used? TIA! -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to iterate through multidimensional space?
Frank Miles wrote: I need to evaluate a complicated function over a multidimensional space as part of an optimization problem. This is a somewhat general problem in which the number of dimensions and the function being evaluated can vary from problem to problem. I've got a working version (with loads of conditionals, and it only works to #dimensions = 10), but I'd like something simpler and clearer and less hard-coded. I've web-searched for some plausible method, but haven't found anything nice. Any recommendations where I should look, or what technique should be used? Not sure this is what you want, but if you have nested for loops -- these can be replaced with itertools.product(): a = [1, 2] b = [10, 20, 30] c = [100] for x in a: ... for y in b: ... for z in c: ... print(x, y, z) ... 1 10 100 1 20 100 1 30 100 2 10 100 2 20 100 2 30 100 for xyz in product(a, b, c): ... print(*xyz) ... 1 10 100 1 20 100 1 30 100 2 10 100 2 20 100 2 30 100 -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic way to iterate through multidimensional space?
On Tue, 05 Aug 2014 20:06:05 +, Frank Miles wrote: I need to evaluate a complicated function over a multidimensional space as part of an optimization problem. This is a somewhat general problem in which the number of dimensions and the function being evaluated can vary from problem to problem. I've got a working version (with loads of conditionals, and it only works to #dimensions = 10), but I'd like something simpler and clearer and less hard-coded. I've web-searched for some plausible method, but haven't found anything nice. Any recommendations where I should look, or what technique should be used? TIA! A should have waited. The answer: itertools.product! very nice -- https://mail.python.org/mailman/listinfo/python-list
