Hi Georg, 2011/4/3 Georg Holzmann <g...@mur.at>
> Hallo! > > Sorry for the maybe stupid question, but is there a way in PyTables to > just import/export big binary (floating point) data into a hdf5 file ? > > I have big audio files (bigger than my memory), manage the data with > PyTables, access parts of them (which really works nice!) and finally > write one output data file. > > However, at the moment I read the binary input data with numpy.fromfile, > which reads the whole file in memory. > Is there something like h5import, which just imports the whole binary > files ? > Or is it also possible to make something similar to an ExternalLink to > binary data ? > I'd try first the h5import venue. If that does not work (because h5import also tries to load the whole binary file in memory), then I'd read your binary files in chunks (by using the regular `file` object in Python) and then add these chunks to an EArray object in PyTables (via the `append()` method). > I also stumbled upon this thread [1], where memmap is suggested. > However, this has the problem that I cannot make larger indezes on 32bit > system ... > (And doesn't this still reserve the whole memory ?) > No, memmap does not reserve the whole memory, at least not initially, but my experience with memmap is that, if you are going to read the whole file, it will end by trying to use all your virtual memory, so I should not go this venue (but you may want to try it anyways). -- Francesc Alted
------------------------------------------------------------------------------ Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf
_______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
