Hi Irek & Richard, thanks for your responds to the previous discussion about developing a generic HDF5 data import module for OpenDX. Below are my comments on that.
I also want to announce the availability of our OpenDXutils package which contains a module ImportHDF5 to read datasets from an HDF5 file into OpenDX. I tried to put in all the features we discussed recently, in the hope that the module will work and be useful for other people as well. Please refer to the following web page for details about where to get the OpenDXutils package from, how to build it, and how to use the ImportHDF5 module (some example networks are provided) http://www.cactuscode.org/VizTools/OpenDX.html Of course, any feedback, hints and suggestions are highly welcome. Ireneusz SZCZESNIAK wrote: > Why not to say that the format of the unigrid data stored in an HDF5 > file should have "origin" and "delta"? In case they are missing, the > default value of "origin" is {0, ... , 0}, and the default value of > "delta" is {{1, 0, ..., 0}, {0, 1, 0, ... , 0}, ..., {0, ... , 0, 1}}. That's exactly what ImportHDF5 implements. > I like the functionality of your module, and I guess these features > must be very useful for you. However, our datasets are very large and > numerous. Therefore we cannot load all of them at once into the > memory to produce an animation. Instead, we have to load one dataset > at a time, produce an image, and then repeat the process. In this way > we can create longer and more accurate animations. Again, that's what ImportHDF5 does. It imports one dataset (or a slab of it) at a time. Setting the stride input appropriately, the module can thus import data even from very large datasets (we tried it on 1500^3 grid size). It's the additional max_index output tab which provides you with the total number of datasets there are in an HDF5 file. Given that information from the import module itself, you can easily build a generic animation network with a sequencer. > > I'd start without these complexities in the hdf5 module, as you can > > do nearly all of this stuff within your dx program. > > I agree with Richard. A module is going to be easier to use. Well, not necessarily. Sure, if people aren't interested in the hyperslabbing features then they don't need to care about them. It's up to the import module to provide sensible defaults for those parameters so that the module does what people expect it to do, without much fiddling around. But it is essential for us to have the hyperslabbing functionality already integrated in the import module (and not having to use the Slab module) because our datasets might simply be too large to fit in memory. > At the moment our module creates a field > either of "float" or "double" depending on the data type of the > dataset. ImportHDF5 can also read integer datasets from HDF5 files. It turns out that the data component of a DX field can also be of integer type. So I just added this feature. > This summer we will be working intensly for roughly a month on our > package. We will let you know of our advances. Please keep us posted > too. Thank you for sharing your ideas! Thanks to you for this discussion ! Let me finally come back to the question of merging our efforts of developing a general HDF5 data import module which fits peoples' needs (for their unigrid data at least), and make it eventually available to the main distribution of OpenDX. Do you think that's possible ? As I see it the biggest issue here isn't of technical nature (we almost agreed on a common file layout) but rather what license should be used for such a module. For your dxhdf5 package you use a special UoC license, Richard has the Academic Free License, and I'm using GPL for the OpenDXutils package. Any chance of finding a solution which works for everyone (not just for us from "Good Old Europe") ? Ciao, Thomas. -- ========================================================================= Thomas Radke Max-Planck-Institute for Gravitational Physics, Albert-Einstein-Institute Am Muehlenberg 1, 14476 Golm, Germany fon +49 331 567-7329 fax +49 331 567-7298 http://www.aei.mpg.de/~tradke "Die USA wollen Irak in 3 Zonen aufteilen: Normal, Super und Diesel." =========================================================================
