Hi,
I am using HDF5 to store a large number (approximately 50000) of
compound datasets composed of a timestamp and a double value (I need a
separate dataset for each item since the timestamps are not
synchronized). Each data is arriving at approximately 100 Hz. I am
running on a 8-core Xeon with 12 Gb of RAM.
Since the data is arriving "on-the-fly", I am using the H5PT
interface. After running for a while, I ran out of memory in my "real"
application so I created a small test program (see at the end of my
message). With this test program, I can easily end up having an
important memory usage (> 4Gb) by adjusting NUM_APPENDS or
NUM_PACKET_TABLES.
After some digging, I realized that while my test program is running,
I get a lot of H5D_chunk_alloc() calls, but no H5D_chunk_xfree() calls
during execution. H5D_chunk_xfree() only seems to get called when I
H5PTclose() the dataset at the end of the program.
I tried using H5Fflush() and played around a bit with various file
properties (sieve buffers, etc.), but I didn't find a way to force
HDF5 to free() the chunk buffers.
I also did "convert" the sample application to using H5D with
extensible datasets and hyperslabs, but I got the same results (no
H5D_chunk_xfree() during runtime).
Any hints on how to make my application run for a long time (without
H5Dwrite() complaining about memory starvation) would be very much
appreciated.
Thanks,
Viktor
====== Example code ======
#include "hdf5.h"
#include "hdf5_hl.h"
#include <stdlib.h>
#include <iostream>
using namespace std;
#include <boost/thread.hpp>
#include <boost/bind.hpp>
#include <boost/format.hpp>
using boost::format;
// Number of tables to create
#define NUM_PACKET_TABLES 10000
// Chunk size
#define CHUNK_SIZE 4096
// Compression level
#define COMPRESSION_LEVEL -1
// Number of H5PTappend() loops (simulating the arrival of data)
#define NUM_APPENDS 100
// Number of elements to pass to H5PTappend() at each loop
#define APPEND_SIZE 128
// Defintion of the compound data type
typedef struct
{
double data;
long long timestamp;
} data_t;
int main(void)
{
hid_t fid;
hid_t ptable[NUM_PACKET_TABLES];
// Data buffer
data_t data[APPEND_SIZE];
// Open file
fid = H5Fcreate("packet.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
// Create compound data type
hid_t datatype = H5Tcreate(H5T_COMPOUND, sizeof(data_t));
H5Tinsert(datatype, "Data", HOFFSET(data_t, data), H5T_NATIVE_DOUBLE);
H5Tinsert(datatype, "Timestamp", HOFFSET(data_t, timestamp),
H5T_NATIVE_LLONG);
// Create packet tables
for (int i = 0; i < NUM_PACKET_TABLES; i++)
ptable[i] = H5PTcreate_fl(fid, str(format("/data.%1%") %
i).c_str(), datatype, (hsize_t) CHUNK_SIZE, COMPRESSION_LEVEL);
// Close data type handle
H5Tclose(datatype);
for (int i = 0; i < NUM_APPENDS; i++)
for (int j = 0; j < NUM_PACKET_TABLES; j++)
// Random data append - we do not care about the values
in the buffer
H5PTappend(ptable[j], (hsize_t) APPEND_SIZE, data);
for (int i = 0; i < NUM_PACKET_TABLES; i++)
H5PTclose(ptable[i]);
// Close file
H5Fclose(fid);
return 0;
}
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
