Re: [Numpy-discussion] fast numpy i/o

2011-06-27 Thread Robert Kern
On Tue, Jun 21, 2011 at 13:35, Christopher Barker chris.bar...@noaa.gov wrote: Robert Kern wrote: https://raw.github.com/numpy/numpy/master/doc/neps/npy-format.txt Just a note. From that doc:     HDF5 is a complicated format that more or less implements     a hierarchical

Re: [Numpy-discussion] fast numpy i/o

2011-06-27 Thread Robert Kern
On Mon, Jun 27, 2011 at 11:17, Derek Homeier de...@astro.physik.uni-goettingen.de wrote: On 21.06.2011, at 8:35PM, Christopher Barker wrote: Robert Kern wrote: https://raw.github.com/numpy/numpy/master/doc/neps/npy-format.txt Just a note. From that doc:     HDF5 is a complicated format

Re: [Numpy-discussion] fast numpy i/o

2011-06-27 Thread Derek Homeier
On 27.06.2011, at 6:36PM, Robert Kern wrote: Some late comments on the note (I was a bit surprised that HDF5 installation seems to be a serious hurdle to many - maybe I've just been profiting from the fink build system for OS X here - but I also was not aware that the current netCDF is

Re: [Numpy-discussion] fast numpy i/o

2011-06-27 Thread Nils Becker
Hi, Finally, the former Scientific.IO NetCDF interface is now part of scipy.io, but I assume it only supports netCDF 3 (the documentation is not specific about that). This might be the easiest option for a portable data format (if Matlab supports it). Yes, it is NetCDF 3. In recent

Re: [Numpy-discussion] fast numpy i/o

2011-06-21 Thread Neal Becker
Neal Becker wrote: I'm wondering what are good choices for fast numpy array serialization? mmap: fast, but I guess not self-describing? hdf5: ? pickle: self-describing, but maybe not fast? others? I think, in addition, that hdf5 is the only one that easily interoperates with matlab?

Re: [Numpy-discussion] fast numpy i/o

2011-06-21 Thread Robert Kern
On Tue, Jun 21, 2011 at 12:49, Neal Becker ndbeck...@gmail.com wrote: I'm wondering what are good choices for fast numpy array serialization? mmap: fast, but I guess not self-describing? hdf5: ? pickle: self-describing, but maybe not fast? others? NPY:

Re: [Numpy-discussion] fast numpy i/o

2011-06-21 Thread Christopher Barker
Neal Becker wrote: I'm wondering what are good choices for fast numpy array serialization? mmap: fast, but I guess not self-describing? hdf5: ? Should be pretty fast, and self describing -- advantage of being a standard. Disadvantage is that it requires an hdf5 library, which can b a pain

Re: [Numpy-discussion] fast numpy i/o

2011-06-21 Thread Christopher Barker
Neal Becker wrote: I'm wondering what are good choices for fast numpy array serialization? mmap: fast, but I guess not self-describing? hdf5: ? pickle: self-describing, but maybe not fast? others? I think, in addition, that hdf5 is the only one that easily interoperates with matlab?

Re: [Numpy-discussion] fast numpy i/o

2011-06-21 Thread Derek Homeier
On 21.06.2011, at 7:58PM, Neal Becker wrote: I think, in addition, that hdf5 is the only one that easily interoperates with matlab? speaking of hdf5, I see: pyhdf5io 0.7 - Python module containing high-level hdf5 load and save functions. h5py 2.0.0 - Read and write HDF5 files from

Re: [Numpy-discussion] fast numpy i/o

2011-06-21 Thread Simon Lyngby Kokkendorff
Hi, I have been using h5py a lot (both on windows and Mac OSX) and can only recommend it- haven't tried the other options though Cheers, Simon On Tue, Jun 21, 2011 at 8:24 PM, Derek Homeier de...@astro.physik.uni-goettingen.de wrote: On 21.06.2011, at 7:58PM, Neal Becker wrote: I

Re: [Numpy-discussion] fast numpy i/o

2011-06-21 Thread Christopher Barker
Robert Kern wrote: https://raw.github.com/numpy/numpy/master/doc/neps/npy-format.txt Just a note. From that doc: HDF5 is a complicated format that more or less implements a hierarchical filesystem-in-a-file. This fact makes satisfying some of the Requirements difficult. To the