On Jul 16, 2012, at 9:13 AM, Daniele Nicolodi wrote:
> On 13/07/2012 14:31, Nicolas wrote:
>> I would like to know if hdf5 is suitable for real-time data logging or
>> not ?
>
>> So, I am considering the possibility to use hdf5 as a back-end for data
>> storage on disk (and numpy/pytable for internal representation). Do you
>> think it is possible to update hdf5 file on at a regular interval from
>> such python binding ?
>
> Independently on the storage format, logging to disc from a real-time thread
> is not a very good idea: accessing to file system storage (and to mass
> storage in general) has not deterministic latencies.
>
> What is recommended to do in this cases is to have a real-time process or
> thread collect the data and store it in an in memory ring-buffer, from where
> it is collected by a non real-time process and written to disc. It is fairly
> trivial to implement a thread safe single reader, single writer ring-buffer
> (an you can find many implementations online).
With this sort of mechanism, HDF5 would be well-suited to Nicolas'
application. We know of users in the financial field who use HDF5 for
high-throughput real-time data recording.
Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
