[Numpy-discussion] Pattern for reading non-simple binary files
Hi, I'm trying to read in a data from a binary-formatted file. I have the data format, (available at: http://www1.ncdc.noaa.gov/pub/data/documentlibrary/tddoc/td7000.pdf if you're really curious), but it's not what I would consider simple, with a lot of different blocks and messages, some that are optional and some that have different formats depending on the data type. My question is, has anyone dealt with data like this using numpy? Have you found a good pattern for how to construct a numpy dtype dynamically to decode the different parts of the file appropriately as you go along? Any insight would be appreciated. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Pattern for reading non-simple binary files
On Fri, Jan 23, 2009 at 15:31, Ryan May rma...@gmail.com wrote: Hi, I'm trying to read in a data from a binary-formatted file. I have the data format, (available at: http://www1.ncdc.noaa.gov/pub/data/documentlibrary/tddoc/td7000.pdf if you're really curious), but it's not what I would consider simple, with a lot of different blocks and messages, some that are optional and some that have different formats depending on the data type. My question is, has anyone dealt with data like this using numpy? Yes! Have you found a good pattern for how to construct a numpy dtype dynamically to decode the different parts of the file appropriately as you go along? I use mmap and create numpy arrays for each block using the ndarray constructor with the appropriate offset parameter. There isn't much of a pattern for constructing the dtypes except to use constructor functions. Good luck! -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Pattern for reading non-simple binary files
Ryan May wrote: Hi, I'm trying to read in a data from a binary-formatted file. I have the data format, (available at: http://www1.ncdc.noaa.gov/pub/data/documentlibrary/tddoc/td7000.pdf if you're really curious), but it's not what I would consider simple, with a lot of different blocks and messages, some that are optional and some that have different formats depending on the data type. My question is, has anyone dealt with data like this using numpy? Have you found a good pattern for how to construct a numpy dtype dynamically to decode the different parts of the file appropriately as you go along? Any insight would be appreciated. Ryan Ryan, http://currents.soest.hawaii.edu/hg/hgwebdir.cgi/pycurrents/file/d7c5c9aac32d/adcp/rdiraw.py#l1 This gives an example of reading several related and rather complex binary file types generated by (oceanographic) acoustic Doppler current profilers. I have not looked at the format you are dealing with, so I don't know if the methods I used are applicable to your case. Eric ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
