Hi.
I did a quick test:
////////////////////////////////////////////////////
//hdf5
/*file_id = H5Fopen(sHD5, H5F_ACC_RDWR, H5P_DEFAULT);
char buf[256];
start = clock();
for (int i=0;i<iNumFrames;++i)
{
int iFrame = i+1;
sprintf(buf, "/dset%d", iFrame);
hid_t dataset_id = H5Dopen(file_id, buf, H5P_DEFAULT);
H5Dread(dataset_id, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL, H5P_DEFAULT,
fPoints);
H5Dclose(dataset_id);
}
printf("hdf read in %d ticks\n", clock()-start);
H5Fclose(file_id);*/
////////////////////////////////////////////////////
//hdf5 - memory
struct stat filestatus;
stat(sHD5, &filestatus);
char *filebuf = new char[filestatus.st_size];
f = fopen(sHD5, "rb");
fread(filebuf, filestatus.st_size, 1, f);
fclose(f);
file_id = H5LTopen_file_image(filebuf, filestatus.st_size, 0);
char buf[256];
start = clock();
for (int i=0;i<iNumFrames;++i)
{
int iFrame = i+1;
sprintf(buf, "/dset%d", iFrame);
hid_t dataset_id = H5Dopen(file_id, buf, H5P_DEFAULT);
H5Dread(dataset_id, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL, H5P_DEFAULT,
fPoints);
H5Dclose(dataset_id);
}
printf("hdf read in %d ticks\n", clock()-start);
H5Fclose(file_id);
delete [] filebuf;
First part (commented out) opens and reads file from disk. Second loads
whole file to filebuf and opens it from there with H5LTopen_file_image.
In both cases measured speed is almost exactly same.
Reading data from data sets is extremely slow. If I read same amount of
data like this:
f = fopen(sMB, "rb");
start = clock();
for (int i=0;i<iNumFrames;++i)
{
fseek(f, 0, SEEK_SET);
fseek(f, sizeof(float) * i * iNumPoints, SEEK_SET);
fread(fPoints, iNumPoints * sizeof(float), 1, f);
}
printf("mb read in %d ticks\n", clock()-start);
fclose(f);
it is almost two orders of magnitude faster. 30 (fread) versus 1444
(hdf5). I did similar test for BDB and results were near 70. I
understand that reading data directly form disk will always be faster
but results for HDF5 are extremely disappointing. I'm not sure if I'm
doing this right.
regards
Adam
On 2012-04-24 23:56, Frank Baker wrote:
Adam,
User documentation for this feature is nearly complete and is currently in
review. Note that the address below is NOT permanent; the docs will be in the
usual place at www.hdfgroup.org/HDF5/doc/ once HDF5 Release 1.8.9 is final.
In the meantime, the docs are TEMPORARILY available here:
Overview:
http://www.hdfgroup.org/HDF5/doc_test/H51.8docs/Advanced/FileImageOperations/HDF5FileImageOperations.pdf
Reference manual entries:
The relevant reference manual entries are all cross-referenced from "See Also"
in the H5PFget_file_image entry:
http://www.hdfgroup.org/HDF5/doc_test/H51.8docs/RM/RM_H5F.html#File-GetFileImage
Note that there is one high-level function that may suit your needs,
H5LTopen_file_iamge. Several low-level H5F and H5P functions are provided for
situations requiring more customization.
Please let me know if you experience any difficulty with these documents.
Thanks!
-- Frank
HDF Documentation Team
[email protected]
On 24 Apr 2012, at 15:38 PM, Adam Wierzchowski wrote:
Excellent. Where can I find more information on this. I don't see anything like
that in hdf5-1.8.9-pre1-RELEASE.txt.
regards
Adam
On 2012-04-24 22:34, Quincey Koziol wrote:
Hi Mark/Adam,
The "file image" feature is available in the upcoming 1.8.9 release,
which has a prerelease candidate available here:
http://www.hdfgroup.uiuc.edu/ftp/pub/outgoing/hdf5/hdf5-1.8.9-pre1/
Quincey
On Apr 24, 2012, at 2:53 PM, Mark Miller wrote:
Ah, that capability also exists but not in a currently released version
of HDF5. I think it is available in HDF5-1.9 series. Maybe someone from
HDF Group could point you in the right direction. Also, have a looksee
here...
https://visitbugs.ornl.gov/attachments/66/load_core_file_driver_from_image_RFC_v04.pdf
Hopefully, Quincy Koziol might be able point you to actual HDF5 version
tarball supporting this.
Mark
On Tue, 2012-04-24 at 12:45 -0700, Adam Wierzchowski wrote:
Hi,
thank you. Exactly what I needed. But what about reading these data. I
need something like:
H5Fcreate...
H5InitializeWithRawBytesFromMemory(void* dataPointer, int
dataLengthInBytes)
regards
Adam
MM> If I am understanding you correctly, you can do that using the 'core'
MM> virtual file driver (VFD). Have a look at...
MM> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore
MM> You need to create a file-access or file-creation properties list object
MM> with appropriate values set for the core VFD and it should behave as you
MM> describe.
MM> Good luck.
MM> Mark
MM> On Tue, 2012-04-24 at 06:55 -0700, Adam Wierzchowski wrote:
Hi,
I have my custom format where I keep all kinds of data: strings, arrays
etc. I'd also like to keep chunk of data which is basically whole hdf
file. Is it possible?
Idea is to keep hdf5 file in memory and dump all that data to my file.
When reading just read all that data to memory and read from it with
hdf5 functions. Problem is that haven't found functions which will allow
me to do that in hdf5 docs.
hope you can help me
regards
Adam
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
[email protected] urgent: [email protected]
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
