Hi,
what's the reason for using a 2D extendable dataset instead of a sequence
of 1D arrays
in a group, using one group per time step? How many particles and time
steps do you
have typically? I assume in your case the number of particles is constant
over time?
Cheers,
Werner
On Wed, 16 Mar 2011 03:52:10 -0500, Yngve Inntjore Levinsen
<[email protected]> wrote:
Dear hierarchical people,
I have currently converted a piece of code from using a simple ascii
format for output into using HDF5. What the code does is at every
iteration dumping some information about particle
energy/trajectory/position to the ascii file (this is a particle
tracking code).
Initially I then did the same with the HDF5 library, having a unlimited
row dimension in a 2D array and using h5extend_f to extend by one
element each time and writing a hyperslab of one row to the file. As
some (perhaps most) of you might have guessed or know already, this was
a rather bad idea. The file (without compression) was about the same
size as the ascii file (but obviously with higher precision), and
reading the file in subsequent analysis was at least an order of
magnitude slower.
I then realized that I probably needed to write less frequently and
rather keeping a semi-large hyperslab in memory. I chose a hyperslab of
1000 rows, but otherwise using the same procedure. This seems to be both
fast and with compression creating quite a bit smaller file. I tried
even larger slabs, but did not see any speed improvement in my initial
testing
My question really was just if there are some recommended ways to do
this? I would imagine I am not the first that want to use HDF5 in this
way, dumping some data at every iteration of a given simulation, without
having to keep it all in memory until the end?
Thanks for all explanations/suggestions/experiences related to this
problem you can provide me so I can make the best design choices in my
program! :)
Cheers,
Yngve
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
