Matt Stumbaugh wrote:
I'm very interested.
Matt, I doubt you're going to learn much from this -- you're no newbie!
As a side note, I've been working with some big datasets in numpy and NetCDF and have found that the product of creating a pickled numpy array versus creating a netcdf file directly is about 3x bigger.
The pickled array is bigger? I"m a bit surprised, I don't think either is compressed.
But since these days I'm having to work on Windows w/ Cygwin and the fact that getting matplotlib made on Cygwin is really tenuous, I've been creating the nc files, using netcdf operators to get hyperslabs and perform basic calcs, then running a conversion to a pickled numpy array to leverage matplotlib (on Windows). Numpy and matplotlib are wonderful aspects of wonderful Python. I've definitely drank the kool-aid!
Have you tried the npy/npz format for arrays? They are pretty handy: numpy.save() numpy.savez() numpy.load()
Someday, I'd like to hear a talk about extending python (wrapping C) and embedding it in C and using ctypes. Though I'd definitely like it to be lite since I'm hardly paper thin on C.
I've been doing some stuff with Cython -- which I think is a better option for most things. But there are folks in SEAPIG that know Cython a lot more that I do (some of the core developers). Maybe a future topic?
-Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception [email protected]
