Hi,
> I'm glad to see other people developing HDF5 readers for OpenDX.
I am also glad. It's good to exchange some ideas.
> Of course, this module should be general enough to fit peoples'
> needs, and that hightly depends on the HDF5 file layout used by the
> various applications which generate HDF5 datafiles.
I belive this is the main problem of our modules. Each of the modules
requires some file layout specific to the software that produced that
file.
> > For plain unigrid data, I think, there exists a de-facto standard
> > that each dataset should have at least an "origin" and a "delta"
> > attribute which will be used to create the positions and
> > connections of the imported DX field.
>
> Which is not necessary, as you can extract the required attributes,
> create positions and replace them as needed in your dx program.
I agree with Thomas, that unigrid data can be stored in an HDF5 file
in a straightforward way (as he wrote above). As Richard points out,
the values of "origin" and "delta" can be extracted and then
incorporated into the filed (though I don't know how). However,
Richard's way of doing must be rather complicated.
Why not to say that the format of the unigrid data stored in an HDF5
file should have "origin" and "delta"? In case they are missing, the
default value of "origin" is {0, ... , 0}, and the default value of
"delta" is {{1, 0, ..., 0}, {0, 1, 0, ... , 0}, ..., {0, ... , 0, 1}}.
Perhaps in the case one has thousands of tiny datasents in an HDF5
file is would not be thrifty to supply "delta" and "origin" to every
dataset. But in most cases the overhead of "delta" and "origin" is
negligible.
> Like Richard, I find it a bit inconvenient for files with multiple
> datasets that a user has to specify each dataset to import
> explicitely by name. One could output the list of dataset names and
> then let the user pick one (or more) out of this list. Our
> ImportHDF5 module takes an index of the dataset to read instead,
> ranging from 0 to the total number of datasets found in the file
> (this information is output on a separate tab). Our datafiles
> usually contain a time series of the same variable, with each
> dataset having a "time" attribute attached to it. If such an attri-
> bute was found during the browsing then the import module would also
> sort the datasets by their time values. Feeding the "max" input tab
> of a Sequencer module with the "total number of datasets"
> information, and connecting the sequencer output tab back to
> ImportHDF5, it is thus trivial to play an animation of all available
> timesteps. To make the dataset selection via indices more general,
> an import module could receive a (list of) attribute name(s) to sort
> the list of datasets ?
I like the functionality of your module, and I guess these features
must be very useful for you. However, our datasets are very large and
numerous. Therefore we cannot load all of them at once into the
memory to produce an animation. Instead, we have to load one dataset
at a time, produce an image, and then repeat the process. In this way
we can create longer and more accurate animations.
> I'd start without these complexities in the hdf5 module, as you can
> do nearly all of this stuff within your dx program.
I agree with Richard. A module is going to be easier to use.
> I want to add this feature to our ImportHDF5 module - so far one can
> specify a single (n-1) slice to read from an n-dimensional dataset.
> ImportHDF5 has a flag to tell it whether floating point datasets
> should be imported in single or double precision. Maybe that's a
> feature you want to add to your module ?
Thanks, that sounds good. At the moment our module creates a field
either of "float" or "double" depending on the data type of the
dataset.
> Also, our ImportHDF5 module can import data from remote HDF5 files
> using the Stream Virtual File Driver of the HDF5 library. And we are
> working on another remote file access driver for HDF5 based on
> GridFtp. In the future we want to do remote visualization of
> large-scale hierar- chical datasets using OpenDX.
That's an excellent feature! That's trully something that is specific
to HDF5. However, now we do not need this and thefore are not
interested in supporting it.
This summer we will be working intensly for roughly a month on our
package. We will let you know of our advances. Please keep us posted
too. Thank you for sharing your ideas!
Best,
Irek
--
********************************************
* Ireneusz SZCZESNIAK - research assistant *
* Tel: +48 (32) 231-73-19, extension 204 *
* http://www.iitis.gliwice.pl/~iszczesniak *
********************************************
____________________________________
The e-mail was checked by antivirus software.
