Thanks a lot Chris for the detailed answer. I had the same doubts about 64bits and multiple cores. I just found out about Gdal and yes, Jeff's grib tools seem to be exactly what I needed.
I'll stay with a fast dual core and 32 bit os and I'll get a 10000 rpm hdd. That should be more than enough especially that this machine will not do anything else but processing this. Sent from my iPhone On Apr 9, 2009, at 10:12 AM, Christopher Barker <chris.bar...@noaa.gov> wrote: Eric Firing wrote: The biggest bottleneck is happening because I'm unpacking grib files to csv files using Degrib in command line. That operation is usually around half an disk speed -- you might want to try SATA RAID 0 (striping) -- I"d get a good hardware vendor's advise in maximizing your disk IO. You can also multi-task that process easily, but if you're disk-bound, that won't help anyway. Instead of going to csv files--which are *very* inefficient to write, store, and then read in again--why not convert directly to netcdf, Or HDF, via PyTables. Or even direct binary numpy arrays, with either fromfile / to file, or, more robustly, with numpy.save and numpy.load. direct numpy-enabled access to the grib files might be even better, eliminating the translation phase entirely. Have you looked into http://www.pyngl.ucar.edu/Nio.shtml? Also, I think GDAL support GRIB, and can directly give you numpy arrays. I have noticed also that on a lower spec AMD desktop this runs faster than on my P4 Intel Laptop, my guess being that the laptop hdd is 5400 rpm and the desktop is 7200 rpm. yup, those laptop hard drives are SLOW -- you culd look into a Solic State drive, if you have some money to spend. Next step is to take all those csv files and make images from them. For this one I haven't dug too deep to see what is happening but it seems to be the other way, using the cpu a lot more while keeping the memory usage high too. mulit-cores aren't going to help here, unless yuo run a few separate processes -- also, how much memory? All 64 bits will buy you is more memory, which you may or may not need. Also, as for Windows 64 bits -- is numpy supported there yet? I'd make sure, there are issues, as there is no MingGW for 64 bit Windows. antonv wrote: I know that using the csv files is very slow but I have no knowledge of working with the netcdf format and I was in a bit of a rush when I wrote this. I will take a look again at it. How would you translate a grib in netcdf? See if degrib supports any binary formats (I now, I'm form NOAA, I should know...). Otherewise yuo could use the hGDAL command-line tools to translate into something else binary that may be easier to deal with. Though it looks like Jeff may have solved this problem for you (One NOAA, Jeff!) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users ------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
