Seems like I had found an issue with header files in hdf5 and it explains why examples files are not compiled property
C:\Program Files (x86)\HDF5 1.8.6\include>grep H5DOpen2 *.h C:\Program Files (x86)\HDF5 1.8.6\include>grep H5Dopen2 *.h H5Dpublic.h:H5_DLL hid_t H5Dopen2(hid_t file_id, const char *name, hid_t dapl_id ); H5public.h: * if((dset = H5Dopen2(file, name)) < 0) H5version.h:#define H5Dopen H5Dopen2 Why are these macro calls inconsistent?? -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of [email protected] Sent: Wed., May. 04, 2011 2:03 PM To: [email protected] Subject: Hdf-forum Digest, Vol 23, Issue 5 Send Hdf-forum mailing list submissions to [email protected] To subscribe or unsubscribe via the World Wide Web, visit http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org or, via email, send a message with subject or body 'help' to [email protected] You can reach the person managing the list at [email protected] When replying, please edit your Subject line so it is more specific than "Re: Contents of Hdf-forum digest..." Today's Topics: 1. Re: How does chunked reading/writing interact with setting hyperslabs in datasets? (Quincey Koziol) 2. Re: chunks of different sizes (Quincey Koziol) 3. Re: behavior of H5Sselect_elements() (Quincey Koziol) 4. Re: Attributes vs Datasets (Quincey Koziol) 5. Re: Problem with getting hdf5 file size (Quincey Koziol) 6. Re: Attributes vs Datasets (Rob Latham) 7. Re: Attributes vs Datasets (Quincey Koziol) 8. Re: Problem with getting hdf5 file size ([email protected]) ---------------------------------------------------------------------- Message: 1 Date: Wed, 4 May 2011 11:06:00 -0500 From: Quincey Koziol <[email protected]> To: HDF Users Discussion List <[email protected]> Subject: Re: [Hdf-forum] How does chunked reading/writing interact with setting hyperslabs in datasets? Message-ID: <[email protected]> Content-Type: text/plain; charset=us-ascii Hi Kent, On May 3, 2011, at 11:20 AM, Williams, Norman K wrote: > OK. > > > I'm not entirely clear how to coordinate different memory and disk > footprints, particularly when using the C++ interface. > > The steps I know about: > > 1. Open the file > 2. Open the DataSet. > 3. Obtain the DataSpace from the Dataset. > 4. Select a hyperslab to read data into a voxel buffer. > > I select a hyperslab from the dataset to read based on the image region > specified by the user. It's a little more complicated than that in ITK, > but that summarizes the process. > > At what stage in the process do I specify a different in-memory dataspace? The voxel buffer is the buffer that you want to describe with the in-memory dataspace. So, you'd define a new dataspace (with H5Screate) that specified the correct dimensions for your buffer. > And how do I specify it's position in the larger on-disk dataspace? You specify a selection in the on-disk dataspace (probably with H5Sselect_all, H5Sselect_hyperslab, H5Sselect_elements, etc) and a selection in the in-memory dataspace (in the same way) and when you call H5Dread, the elements are transferred from the on-disk dataset into the memory buffer. > If there's example code in the manual, just point me there. I think Binh-Minh already pointed you to the C++ examples, and you can search for H5Sselect_hyperslab in the 'examples' subdirectory of the HDF5 distribution. Quincey > On 5/3/11 10:56 AM, "Quincey Koziol" <[email protected]> wrote: > >> Hi Kent, >> >> On May 3, 2011, at 10:16 AM, Williams, Norman K wrote: >> >>> I am using the H5Sselect_hyperslab method for both reading and writing >>> already. Additionally when I write out an image, I 'chunk' the output >>> and >>> compress it. >>> >>> >>> The question I have is what I can do to minimize the memory footprint. >>> Based on my reading in the documentation, the Hyperslab interface >>> actually >>> scatters/gathers to/from an in-memory dataset, leading me to believe the >>> entire dataset will be allocated in system memory. >>> >>> So the question is how would I use HDF5 in such a way as to minimize the >>> memory footprint in this context? >> >> You are allowed to create a memory dataspace that is different from >> the dataset's dataspace in the file. That will allow you to tune the >> memory footprint. >> > > > > ________________________________ > Notice: This UI Health Care e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential and may be legally privileged. If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited. Please reply to the sender that you have received the message in error, then delete it. Thank you. > ________________________________ > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org ------------------------------ Message: 2 Date: Wed, 4 May 2011 11:08:00 -0500 From: Quincey Koziol <[email protected]> To: HDF Users Discussion List <[email protected]> Subject: Re: [Hdf-forum] chunks of different sizes Message-ID: <[email protected]> Content-Type: text/plain; charset=us-ascii Hi Eduardo, On May 3, 2011, at 12:36 PM, Eduardo wrote: >> Different chunk sizes would be interesting to avoid IO contention when >> the hyperslabs are not regular. > > So, if my hyperslabs are not regular (similarly to my previous > example), does my application suffer IO contention? I mean, should I > avoid irregular hyperslabs, even if there is no overlap or gaps? Yes, independent access I/O performance will be best when there is a direct mapping of each process's selection to 1+ chunks and those chunks aren't accessed by other processes. Enabling collective I/O access may reduce or eliminate the penalty for accessing one chunk with multiple processes however. Quincey ------------------------------ Message: 3 Date: Wed, 4 May 2011 11:09:25 -0500 From: Quincey Koziol <[email protected]> To: HDF Users Discussion List <[email protected]> Cc: Prabhat <[email protected]> Subject: Re: [Hdf-forum] behavior of H5Sselect_elements() Message-ID: <[email protected]> Content-Type: text/plain; charset=us-ascii Hi Mark, On May 3, 2011, at 4:22 PM, Mark Howison wrote: > Hi all, > > We use this call in H5hut to allow the user to do a point selection on > a dataset. Previously, I thought it worked like in this example: > > 1) I call H5Sselect_elements() with the H5S_SELECT_SET operation, an > "coord" array of [0,3,6,9] and num_elements=4 > 2) I do a H5Dread to a buffer with enough space for 4 elements, and I > get back a *packed* array with elements 0,3,6,9 and nothing in between > > Recently though our regression test for this in H5hut failed with > 1.8.5 and 1.8.6, and when I looked closer I realized that H5Dread is > giving me a sparsely packed array that wants to be 10 elements long > with only 0,3,6,9 filled in and the others left untouched. > > Did the behavior of H5Sselect_elements() change in a recent release, > or has there always been something wrong with our regression test, and > I never should have gotten back a packed array from H5Dread? Nope, H5Sselect_elements has never returned a packed array. I'd lean toward an incorrect regression test. (Although it's always possible to have a bug in the HDF5 library, also) Quincey ------------------------------ Message: 4 Date: Wed, 4 May 2011 11:13:08 -0500 From: Quincey Koziol <[email protected]> To: HDF Users Discussion List <[email protected]> Subject: Re: [Hdf-forum] Attributes vs Datasets Message-ID: <[email protected]> Content-Type: text/plain; charset=us-ascii Hi John, On May 4, 2011, at 2:10 AM, Biddiscombe, John A. wrote: > If I know that I want to attach a couple of arrays as attributes, and these arrays may be say 1024 elements of double precision max. (say 8kB). The datasets themselves might be xGB. > > 1) Can I tell hdf5 to reserve a minimum size in the object header so that I know in advance that the attributes will fit (or does the object header only contain pointers to other structures anyway) This feature isn't available through the public API currently. (I have thought about exposing the internal part of the library that would allow it, but it seemed like it would only be used by a very small portion of the user base) > 2) Is there any performance advantage in terms of fewer metadata transaction to using attributes over datasets in a parallel IO context. (file is written in parallel, only rank zero needs to actually write the attributes) Hmm, as usual, it depends. :-) Are all the processes going to be accessing the attribute? If not, you could create an attribute with an object reference to an auxiliary dataset, and then read that in dataset in when needed. Quincey > Question motivated by .... > > From the docs ... > "The HDF5 format and I/O library are designed with the assumption that attributes are small datasets. They are always stored in the object header of the object they are attached to. Because of this, large datasets should not be stored as attributes. How large is "large" is not defined by the library and is up to the user's interpretation. (Large datasets with metadata can be stored as supplemental datasets in a group with the primary dataset.)" > > Thanks > > JB > > -- > John Biddiscombe, email:biddisco @ cscs.ch > http://www.cscs.ch/ > CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07 > Via Cantonale, 6928 Manno, Switzerland | Fax: +41 (91) 610.82.82 > > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org ------------------------------ Message: 5 Date: Wed, 4 May 2011 11:16:29 -0500 From: Quincey Koziol <[email protected]> To: HDF Users Discussion List <[email protected]> Subject: Re: [Hdf-forum] Problem with getting hdf5 file size Message-ID: <[email protected]> Content-Type: text/plain; charset="us-ascii" Hi Daniel, On May 4, 2011, at 3:13 AM, Sentenac Daniel wrote: > Hello, > I am currently doing a very small test to retreive the size of a just created HDF5 file using the function > H5Fget_filesize and cannot understand why the size returned is actually different from the real one. > > Here is the simple example piece of code: > > hsize_t size; > hid_t file_id = H5Fcreate("file.hdf5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT); > H5Fget_filesize( file_id, &size ); > > The real size of the file appears to be 800 bytes, while the size returned fromH5Fget_filesize is 2144 bytes. The library allocates a 2KB buffer to sub-allocate metadata from (in addition to the ~100 byte superblock for the file), but then truncates the file's size down to the actual amount of data used in the file, when the file is closed. > I would like to use the function H5Fget_vfd_handle to get a (char*) buffer copy of the file. Can someone help me to do that correctly ? I would recommend against this sort of mucking with the bytes - the HDF5 library does a lot of work behind the scenes and you probably won't get a coherent picture of the file by just looking at the bytes. That said, why do you want to do this? Possibly the "core" VFD (H5Pset_fapl_core) would meet your needs? Quincey -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/2 0110504/1a06b74b/attachment-0001.html> ------------------------------ Message: 6 Date: Wed, 4 May 2011 11:21:30 -0500 From: Rob Latham <[email protected]> To: HDF Users Discussion List <[email protected]> Subject: Re: [Hdf-forum] Attributes vs Datasets Message-ID: <[email protected]> Content-Type: text/plain; charset=us-ascii On Wed, May 04, 2011 at 11:13:08AM -0500, Quincey Koziol wrote: > > 1) Can I tell hdf5 to reserve a minimum size in the object header so that I know in advance that the attributes will fit (or does the object header only contain pointers to other structures anyway) > > This feature isn't available through the public API currently. (I have thought about exposing the internal part of the library that would allow it, but it seemed like it would only be used by a very small portion of the user base) Seems like something the NetCDF-4 folks could use to good effect, if they don't already. After exiting NetCDF define mode, the size of the attributes and objects will be known. NetCDF callers are familiar with the potential pain of re-entering define mode. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA ------------------------------ Message: 7 Date: Wed, 4 May 2011 11:32:28 -0500 From: Quincey Koziol <[email protected]> To: HDF Users Discussion List <[email protected]> Subject: Re: [Hdf-forum] Attributes vs Datasets Message-ID: <[email protected]> Content-Type: text/plain; charset=us-ascii On May 4, 2011, at 11:21 AM, Rob Latham wrote: > On Wed, May 04, 2011 at 11:13:08AM -0500, Quincey Koziol wrote: >>> 1) Can I tell hdf5 to reserve a minimum size in the object header so that I know in advance that the attributes will fit (or does the object header only contain pointers to other structures anyway) >> >> This feature isn't available through the public API currently. (I have thought about exposing the internal part of the library that would allow it, but it seemed like it would only be used by a very small portion of the user base) > > Seems like something the NetCDF-4 folks could use to good effect, if > they don't already. After exiting NetCDF define mode, the size of the > attributes and objects will be known. NetCDF callers are familiar with > the potential pain of re-entering define mode. Ah, good point. (Although adding new attributes to an existing HDF5 object can be done at any point, without any of the pain that netCDF has for re-entering define mode) Quincey ------------------------------ Message: 8 Date: 4 May 2011 18:02:45 -0000 From: [email protected] To: [email protected] Subject: Re: [Hdf-forum] Problem with getting hdf5 file size Message-ID: <[email protected]> Content-Type: text/plain; charset="us-ascii" An HTML attachment was scrubbed... URL: <http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/2 0110504/91d6d1a7/attachment.html> ------------------------------ _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org End of Hdf-forum Digest, Vol 23, Issue 5 **************************************** _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
