There are a number of ways to do this sort of thing. The packet tables as mentioned in another response is one thing. Packet tables have the advantage that you don't need a priori knowledge of how many records will fit in a table. There is, however, a performance hit for this.
You also have the option of either using compound types (structures) or scalar types (if all of your measurements are the same type, e.g. all double-precision floating point, or whatever). There can be a performance hit with compound types relative to scalars as well. I think in terms of performance, the options are as follows (roughly lowest overhead to highest, it would be a good idea to thoroughly test a number of different options to see which one gives you the best balance of performance and capability): 1) pre-sized dataset with scalar types 2) pre-sized dataset with compound types 3) packet table with scalar types 4) packet table with compound types In my own work, I use pre-sized datasets with compound types. Our data are not homogeneous, so the compound types are pretty much required. The pre-sized datasets affords us the ability to actively index the data on read and write, which is of particular value on the read-side where knowing exactly where to find the data you want is highly valuable. If, on the other hand, you just need to store data that you're never going to do partial reads of, packet tables are easier to implement. Daniele Nicolodi wrote:
Hello, I'm really new to HDF5 and I'm trying to understand what would be the best structure for "real time" logging of experimental data. More in detail, I have a real time acquisition system that collects data from multiple sources at a constant rate. Data logging is now performed putting the data in a ring buffer and having a consumer non real time process that takes the data from the ring buffer and writes them into an ASCII file. The file is rotated after a certain number of samples have been written. Loading the data from ASCII file is however slow, and ASCII files are not the best option for storing metadata. There fore I would like to switch to HDF5 as storage format. However, it is not clear to me what would be the most suitable structure for my files. I imagine I need a structure to which I can append data, but it is not clear to me if I should handle my data as a collection of columns (one for each channel, plus one for the timestamp) or as a collection of records (one record for each acquisition cycle) or if a table like structure would be better. Thank you very much for any hint on this topic. Best regards,
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
