Re: [Numpy-discussion] Efficient reading of binary data

2008-04-04 Thread Sebastian Haase
On Fri, Apr 4, 2008 at 2:14 AM, Nicolas Bigaouette [EMAIL PROTECTED] wrote: 2008/4/3, Robert Kern [EMAIL PROTECTED]: On Thu, Apr 3, 2008 at 6:53 PM, Nicolas Bigaouette [EMAIL PROTECTED] wrote: Thanx for the fast response Robert ;) I changed my code to use the slice: E =

Re: [Numpy-discussion] Efficient reading of binary data

2008-04-04 Thread Sebastian Haase
On Fri, Apr 4, 2008 at 11:33 AM, Jarrod Millman [EMAIL PROTECTED] wrote: On Fri, Apr 4, 2008 at 1:50 AM, Sebastian Haase [EMAIL PROTECTED] wrote: Hi, Accidentally I'm exactly trying to do the same thing right now . What is the best way of memmapping into a file that is already

Re: [Numpy-discussion] Efficient reading of binary data

2008-04-04 Thread Jarrod Millman
On Fri, Apr 4, 2008 at 1:50 AM, Sebastian Haase [EMAIL PROTECTED] wrote: Hi, Accidentally I'm exactly trying to do the same thing right now . What is the best way of memmapping into a file that is already open !? I have to read some text (header info) off the beginning of the file

Re: [Numpy-discussion] Efficient reading of binary data

2008-04-04 Thread Christopher Barker
Nicolas Bigaouette wrote: So the next step would be to only read the needed data from the binary file... You've gotten some suggestions, but another option is to use file.seek(0 to get where your data is, and numpy.fromfile() from there. -CHB -- Christopher Barker, Ph.D. Oceanographer

Re: [Numpy-discussion] Efficient reading of binary data

2008-04-03 Thread Robert Kern
On Thu, Apr 3, 2008 at 3:30 PM, Nicolas Bigaouette [EMAIL PROTECTED] wrote: Hi, I have a C program which outputs large (~GB) files. It is a simple binary dump of an array of structure containing 9 doubles. You can see this as a double 1D array of size 9*Stot (Stot being the allocated size of

Re: [Numpy-discussion] Efficient reading of binary data

2008-04-03 Thread Nicolas Bigaouette
Thanx for the fast response Robert ;) I changed my code to use the slice: E = data[6::9] It is indeed faster and less eat less memory. Great. Thanx for the endiannes! I knew there was something like this ;) I suspect that, in 'f8', f means float and 8 means 8 bytes? From some benchmarks, I see

Re: [Numpy-discussion] Efficient reading of binary data

2008-04-03 Thread Robert Kern
On Thu, Apr 3, 2008 at 6:53 PM, Nicolas Bigaouette [EMAIL PROTECTED] wrote: Thanx for the fast response Robert ;) I changed my code to use the slice: E = data[6::9]It is indeed faster and less eat less memory. Great. Thanx for the endiannes! I knew there was something like this ;) I suspect

Re: [Numpy-discussion] Efficient reading of binary data

2008-04-03 Thread Nicolas Bigaouette
2008/4/3, Robert Kern [EMAIL PROTECTED]: On Thu, Apr 3, 2008 at 6:53 PM, Nicolas Bigaouette [EMAIL PROTECTED] wrote: Thanx for the fast response Robert ;) I changed my code to use the slice: E = data[6::9]It is indeed faster and less eat less memory. Great. Thanx for the endiannes!