Hello Pauli,
A Sunday 02 December 2007, Pauli Virtanen escrigué:
> Hi Francesc & Quincey,
>
> I was a bit busy earlier, but now I have a self-contained C snippet
> demonstrating the large memory usage when writig to a HDF5 dataset
> using a chunk shape "incompatible" with the hyperslab shape. The
> relevant H5* call sequences should be quite equivalent to what's
> going on in the Python test program with Pytables 2.0.2.
>
> I have tested this now with hdf5 1.6.5 and 1.8.0beta5, and for both
> versions more memory than 1 GB is allocated when H5Dwrite is called.
Ups, I've ended with a similar program and send it to the
[EMAIL PROTECTED] list past Saturday. I'm attaching my own version
(which is pretty similar to yours). Sorry for not sending you a copy
of my previous message, because it could saved you some work :-/
Thanks for your collaboration anyway!
--
>0,0< Francesc Altet http://www.carabos.com/
V V Cárabos Coop. V. Enjoy Data
"-"
#include "hdf5.h"
#define H5FILE_NAME "write-bug.h5"
#define DATASETNAME "DoubleArray"
/* define N 600 */ /* Try this for seg fault */
#define N 60
#define M 5
#define RANK 6
int
main (void)
{
hid_t file, dataset; /* file and dataset handles */
hid_t datatype, dataspace; /* handles */
hid_t mem_space, plist;
hsize_t dimsf[] = {2,2,N,N,M,M};
hsize_t dimsc[] = {1,1,1,6,M,M};
hsize_t start[] = {0,0,0,0,0,0};
hsize_t step[] = {1,1,1,1,1,1};
hsize_t count[] = {2,2,N,N,1,1};
herr_t status;
double data[2][2][N][N][1][1]; /* data to write */
int i, j;
/*
* Create a new file using H5F_ACC_TRUNC access,
* default file creation properties, and default file
* access properties.
*/
file = H5Fcreate(H5FILE_NAME, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
/*
* Describe the size of the array and create the data space for fixed
* size dataset.
*/
dataspace = H5Screate_simple(RANK, dimsf, NULL);
/* Set the chunkshape */
plist = H5Pcreate(H5P_DATASET_CREATE);
H5Pset_chunk(plist, RANK, dimsc);
/*
* Define datatype for the data in the file.
* We will store little endian DOUBLE numbers.
*/
datatype = H5Tcopy(H5T_NATIVE_DOUBLE);
status = H5Tset_order(datatype, H5T_ORDER_LE);
/*
* Create a new dataset within the file using defined dataspace and
* datatype and default dataset creation properties.
*/
dataset = H5Dcreate(file, DATASETNAME, datatype, dataspace, plist);
/* Fill the dataset starting with trailing dimensions first */
for (j = 0; j < M; j++) {
start[4] = j;
for (i = 0; i < M; i++) {
start[5] = i;
/* Create a simple memory data space */
mem_space = H5Screate_simple(RANK, count, NULL);
/* Define a hyperslab in the dataset */
H5Sselect_hyperslab(dataspace, H5S_SELECT_SET,
start, step, count, NULL);
/* Write the data to the dataset using default transfer properties */
status = H5Dwrite(dataset, H5T_NATIVE_INT, mem_space, dataspace,
H5P_DEFAULT, data);
}
}
/*
* Close/release resources.
*/
H5Sclose(mem_space);
H5Sclose(dataspace);
H5Tclose(datatype);
H5Dclose(dataset);
H5Fclose(file);
return 0;
}
-------------------------------------------------------------------------
SF.Net email is sponsored by: The Future of Linux Business White Paper
from Novell. From the desktop to the data center, Linux is going
mainstream. Let it simplify your IT future.
http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
