Re: [Numpy-discussion] puzzle: generate index with many ranges
Beautiful! That should do the trick. Now let's see how this performs against the list comprehension... Thanks a lot! Raik Rick White wrote: Here's a technique that works: Python 2.4.2 (#5, Nov 21 2005, 23:08:11) [GCC 4.0.0 20041026 (Apple Computer, Inc. build 4061)] on darwin Type help, copyright, credits or license for more information. import numpy as np a = np.array([0,4,0,11]) b = np.array([-1,11,4,15]) rangelen = b-a+1 cumlen = rangelen.cumsum() c = np.arange(cumlen[-1],dtype=np.int32) c += np.repeat(a[1:]-c[cumlen[0:-1]], rangelen[1:]) print c [ 4 5 6 7 8 9 10 11 0 1 2 3 4 11 12 13 14 15] The basic idea is that the difference of your desired output from a simple range is an array with a bunch of constant values appended together, and that is what repeat() does. I'm assuming that you'll never have b a. Notice the slight ugliness of prepending the elements at the beginning so that the cumsum starts with zero. (Maybe there is a cleaner way to do that.) This does create a second array (via the repeat) that is the same length as the result. If that uses too much memory, you could break up the repeat and update of c into segments using a loop. (You wouldn't need a loop for every a,b element -- do a bunch in each iteration.) -- Rick Raik Gruenberg wrote: Hi there, perhaps someone has a bright idea for this one: I want to concatenate ranges of numbers into a single array (for indexing). So I have generated an array a with starting positions, for example: a = [4, 0, 11] I have an array b with stop positions: b = [11, 4, 15] and I would like to generate an index array that takes 4..11, then 0..4, then 11..15. In reality, a and b have 1+ elements and the arrays to be sliced are very large so I want to avoid any for loops etc. Any idea how this could be done? I thought some combination of *repeat* and adding of *arange* should do the trick but just cannot nail it down. Thanks in advance for any hints! Greetings, Raik ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Dr. Raik Gruenberg http://www.raiks.de/contact.html ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] puzzle: generate index with many ranges
Hi there, perhaps someone has a bright idea for this one: I want to concatenate ranges of numbers into a single array (for indexing). So I have generated an array a with starting positions, for example: a = [4, 0, 11] I have an array b with stop positions: b = [11, 4, 15] and I would like to generate an index array that takes 4..11, then 0..4, then 11..15. In reality, a and b have 1+ elements and the arrays to be sliced are very large so I want to avoid any for loops etc. Any idea how this could be done? I thought some combination of *repeat* and adding of *arange* should do the trick but just cannot nail it down. Thanks in advance for any hints! Greetings, Raik ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] puzzle: generate index with many ranges
Raik Gruenberg wrote: Hi there, perhaps someone has a bright idea for this one: I want to concatenate ranges of numbers into a single array (for indexing). So I have generated an array a with starting positions, for example: a = [4, 0, 11] I have an array b with stop positions: b = [11, 4, 15] and I would like to generate an index array that takes 4..11, then 0..4, then 11..15. Does this help? Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] on win32 Type help, copyright, credits or license for more information. a = [4, 0, 11] b = [11, 4, 15] zip(a,b) [(4, 11), (0, 4), (11, 15)] Apologies if I'm stating the obvious. -- jv In reality, a and b have 1+ elements and the arrays to be sliced are very large so I want to avoid any for loops etc. Any idea how this could be done? I thought some combination of *repeat* and adding of *arange* should do the trick but just cannot nail it down. Thanks in advance for any hints! Greetings, Raik ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] puzzle: generate index with many ranges
Jim Vickroy wrote: Raik Gruenberg wrote: Hi there, perhaps someone has a bright idea for this one: I want to concatenate ranges of numbers into a single array (for indexing). So I have generated an array a with starting positions, for example: a = [4, 0, 11] I have an array b with stop positions: b = [11, 4, 15] and I would like to generate an index array that takes 4..11, then 0..4, then 11..15. Does this help? Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] on win32 Type help, copyright, credits or license for more information. a = [4, 0, 11] b = [11, 4, 15] zip(a,b) [(4, 11), (0, 4), (11, 15)] Mhm, I got this far. But how do I get from here to a single index array [ 4, 5, 6, ... 10, 0, 1, 2, 3, 11, 12, 13, 14 ] ? Greetings Raik Apologies if I'm stating the obvious. -- jv In reality, a and b have 1+ elements and the arrays to be sliced are very large so I want to avoid any for loops etc. Any idea how this could be done? I thought some combination of *repeat* and adding of *arange* should do the trick but just cannot nail it down. Thanks in advance for any hints! Greetings, Raik ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Dr. Raik Gruenberg http://www.raiks.de/contact.html ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] puzzle: generate index with many ranges
Raik Gruenberg wrote: Jim Vickroy wrote: Raik Gruenberg wrote: Hi there, perhaps someone has a bright idea for this one: I want to concatenate ranges of numbers into a single array (for indexing). So I have generated an array a with starting positions, for example: a = [4, 0, 11] I have an array b with stop positions: b = [11, 4, 15] and I would like to generate an index array that takes 4..11, then 0..4, then 11..15. Does this help? Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] on win32 Type help, copyright, credits or license for more information. a = [4, 0, 11] b = [11, 4, 15] zip(a,b) [(4, 11), (0, 4), (11, 15)] Mhm, I got this far. But how do I get from here to a single index array [ 4, 5, 6, ... 10, 0, 1, 2, 3, 11, 12, 13, 14 ] ? not sure I understand your goal ... is this what you want: [range(i,j) for i,j in zip(a,b)] [[4, 5, 6, 7, 8, 9, 10], [0, 1, 2, 3], [11, 12, 13, 14]] Greetings Raik Apologies if I'm stating the obvious. -- jv In reality, a and b have 1+ elements and the arrays to be sliced are very large so I want to avoid any for loops etc. Any idea how this could be done? I thought some combination of *repeat* and adding of *arange* should do the trick but just cannot nail it down. Thanks in advance for any hints! Greetings, Raik ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] puzzle: generate index with many ranges
On Jan 30, 2009, at 1:11 PM, Raik Gruenberg wrote: Mhm, I got this far. But how do I get from here to a single index array [ 4, 5, 6, ... 10, 0, 1, 2, 3, 11, 12, 13, 14 ] ? np.concatenate([np.arange(aa,bb) for (aa,bb) in zip(a,b)]) ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] puzzle: generate index with many ranges
Pierre GM wrote: On Jan 30, 2009, at 1:11 PM, Raik Gruenberg wrote: Mhm, I got this far. But how do I get from here to a single index array [ 4, 5, 6, ... 10, 0, 1, 2, 3, 11, 12, 13, 14 ] ? np.concatenate([np.arange(aa,bb) for (aa,bb) in zip(a,b)]) exactly! Now, the question was, is there a way to do this only using numpy functions (sum, repeat, ...), that means without any python for loop? Sorry about being so insistent on this one but, in my experience, eliminating those for loops makes a huge difference in terms of speed. The zip is probably also quite costly on a very large data set. Thanks! Raik ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion -- Dr. Raik Gruenberg http://www.raiks.de/contact.html ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] puzzle: generate index with many ranges
On Jan 30, 2009, at 1:53 PM, Raik Gruenberg wrote: Pierre GM wrote: On Jan 30, 2009, at 1:11 PM, Raik Gruenberg wrote: Mhm, I got this far. But how do I get from here to a single index array [ 4, 5, 6, ... 10, 0, 1, 2, 3, 11, 12, 13, 14 ] ? np.concatenate([np.arange(aa,bb) for (aa,bb) in zip(a,b)]) exactly! Now, the question was, is there a way to do this only using numpy functions (sum, repeat, ...), that means without any python for loop? Can't really see it right now. Make np.arange(max(b)) and take the slices you need ? But you still have to look in 2 arrays to find the beginning and end of slices, so... Sorry about being so insistent on this one but, in my experience, eliminating those for loops makes a huge difference in terms of speed. The zip is probably also quite costly on a very large data set. yeah, but it's in a list comprehension, which may make things a tad faster. If you prefer, use itertools.izip instead of zip, but I wonder where the advantage would be. Anyway, are you sure this particular part is your bottleneck ? You know the saying about premature optimization... ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] puzzle: generate index with many ranges
Here's a technique that works: Python 2.4.2 (#5, Nov 21 2005, 23:08:11) [GCC 4.0.0 20041026 (Apple Computer, Inc. build 4061)] on darwin Type help, copyright, credits or license for more information. import numpy as np a = np.array([0,4,0,11]) b = np.array([-1,11,4,15]) rangelen = b-a+1 cumlen = rangelen.cumsum() c = np.arange(cumlen[-1],dtype=np.int32) c += np.repeat(a[1:]-c[cumlen[0:-1]], rangelen[1:]) print c [ 4 5 6 7 8 9 10 11 0 1 2 3 4 11 12 13 14 15] The basic idea is that the difference of your desired output from a simple range is an array with a bunch of constant values appended together, and that is what repeat() does. I'm assuming that you'll never have b a. Notice the slight ugliness of prepending the elements at the beginning so that the cumsum starts with zero. (Maybe there is a cleaner way to do that.) This does create a second array (via the repeat) that is the same length as the result. If that uses too much memory, you could break up the repeat and update of c into segments using a loop. (You wouldn't need a loop for every a,b element -- do a bunch in each iteration.) -- Rick Raik Gruenberg wrote: Hi there, perhaps someone has a bright idea for this one: I want to concatenate ranges of numbers into a single array (for indexing). So I have generated an array a with starting positions, for example: a = [4, 0, 11] I have an array b with stop positions: b = [11, 4, 15] and I would like to generate an index array that takes 4..11, then 0..4, then 11..15. In reality, a and b have 1+ elements and the arrays to be sliced are very large so I want to avoid any for loops etc. Any idea how this could be done? I thought some combination of *repeat* and adding of *arange* should do the trick but just cannot nail it down. Thanks in advance for any hints! Greetings, Raik ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
