Eric Firing wrote: >> The biggest bottleneck is happening because I'm unpacking grib files to csv >> files using Degrib in command line. That operation is usually around half an
disk speed -- you might want to try SATA RAID 0 (striping) -- I"d get a good hardware vendor's advise in maximizing your disk IO. You can also multi-task that process easily, but if you're disk-bound, that won't help anyway. > Instead of going to csv files--which are *very* inefficient to write, > store, and then read in again--why not convert directly to netcdf, Or HDF, via PyTables. Or even direct binary numpy arrays, with either fromfile / to file, or, more robustly, with numpy.save and numpy.load. > direct numpy-enabled access to the grib files might be > even better, eliminating the translation phase entirely. Have you > looked into http://www.pyngl.ucar.edu/Nio.shtml? Also, I think GDAL support GRIB, and can directly give you numpy arrays. >> I have noticed also that on a lower spec AMD desktop this runs >> faster than on my P4 Intel Laptop, my guess being that the laptop hdd is >> 5400 rpm and the desktop is 7200 rpm. yup, those laptop hard drives are SLOW -- you culd look into a Solic State drive, if you have some money to spend. >> Next step is to take all those csv files and make images from them. For this >> one I haven't dug too deep to see what is happening but it seems to be the >> other way, using the cpu a lot more while keeping the memory usage high too. mulit-cores aren't going to help here, unless yuo run a few separate processes -- also, how much memory? All 64 bits will buy you is more memory, which you may or may not need. Also, as for Windows 64 bits -- is numpy supported there yet? I'd make sure, there are issues, as there is no MingGW for 64 bit Windows. antonv wrote: > I know that using the csv files is very slow but I have no knowledge of > working with the netcdf format and I was in a bit of a rush when I wrote > this. I will take a look again at it. How would you translate a grib in > netcdf? See if degrib supports any binary formats (I now, I'm form NOAA, I should know...). Otherewise yuo could use the hGDAL command-line tools to translate into something else binary that may be easier to deal with. Though it looks like Jeff may have solved this problem for you (One NOAA, Jeff!) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ------------------------------------------------------------------------------ This SF.net email is sponsored by: High Quality Requirements in a Collaborative Environment. Download a free trial of Rational Requirements Composer Now! http://p.sf.net/sfu/www-ibm-com _______________________________________________ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
