I think the comment in h5test.c is misleading.
The documentation for H5Pset_fapl_core says the inc argument is the
number of BYTES, not MEGAbytes.
I'd bet the initial malloc is failing for reading your whole file.
Sorry. Didn't expect that. But, I was thinking in terms of 64 bits and
30 gigs 5 bits too many for a 32 bit number. So, if you are running in
truly 64 bit, then I am completely puzzled. Otherwise, if you are
running 32 bit, then I think the problem has presented itself; we can't
address 30 gig buffer in a 32 bit application.
I'd suggest looking at some of the other properties such as
H5Pset_small_data_block_size
HP5set_buffer (used during read/xform only)
H5Pset_cache (relevant only for chunked datasets)
H5Pset_meta_block_size( hid_t fapl_id, hsize_t size )
and see if setting any/all of these to large numbers; 100's of megabytes
has any positive impact.
Mark
On Wed, 2010-03-03 at 09:30, Thorben Kröger wrote:
> On Wednesday 03 March 2010 18:17:31 Mark Miller wrote:
> > I think the '5' in the HP5set_fapl_core means to allocate only 5 more
> > bytes each time it needs to allocate. I think that is too small and
> > maybe a problem since its an odd number and not a power of 2.
> >
> > Although I would not necessarily expect that to have failed in the way
> > you are seeing either.
> >
> > Try '5<<20' instead of '5'
>
> I found this in h5test.c, function h5_fileaccess()
>
> /* In-core temporary file with 1MB increment */
> if (H5Pset_fapl_core(fapl, (size_t)1, TRUE)<0) return -1;
>
> Anyway, using a number like 5<<20 did not make a difference, it still
> aborts...
>
> Cheers,
> Thorben
>
> >
> > Other than that, I don't see anything wrong with what your are doing.
> > Except that maybe allocating the 30 gigabyte buffer in HDF5 to read your
> > file into core is failing due to too large a request. I guess that is
> > possible. To see if that is happening, if your HDF5 installation was
> > compiled with tracing/debugging features turned on, then try setting
> > HDF5_DEBUG=all in your enviornment where you are running your
> > application.
> >
> > Mark
> >
> > On Wed, 2010-03-03 at 09:07, Thorben Kröger wrote:
> > > > > On Wednesday 03 March 2010 16:47:20 Mark Miller wrote:
> > > > > > Hello Thorben,
> > > > > >
> > > > > > However, did you in fact try using HDF5-1.8.4
> > > > > > and reading the file with the CORE vfd (H5Pset_fapl_core)
> > >
> > > instead of opening my file like this:
> > >
> > > hdf5geometryFile_ = H5Fopen(filename.c_str(), H5F_ACC_RDONLY,
> > > H5P_DEFAULT);
> > >
> > > I am now doing this:
> > >
> > > hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
> > > if(fapl < 0) {throw std::runtime_error("fapl < 0");}
> > > hid_t ret = H5Pset_fapl_core(fapl, 5/*MB?*/, 0 /*no backing store*/);
> > > if(ret < 0) {throw std::runtime_error("could not create fapl core");}
> > > hdf5geometryFile_ = H5Fopen(filename.c_str(), H5F_ACC_RDONLY, fapl);
> > >
> > > But using the second way, it cannot open any of my datasets:
> > >
> > > HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
> > > #000: H5D.c line 321 in H5Dopen2(): not found
> > >
> > > major: Dataset
> > > minor: Object not found
> > >
> > > #001: H5Gloc.c line 468 in H5G_loc_find(): can't find object
> > >
> > > major: Symbol table
> > > minor: Object not found
> > >
> > > #002: H5Gtraverse.c line 877 in H5G_traverse(): internal path traversal
> > >
> > > failed
> > >
> > > major: Symbol table
> > > minor: Object not found
> > >
> > > #003: H5Gtraverse.c line 665 in H5G_traverse_real(): can't look up
> > > component
> > >
> > > major: Symbol table
> > > minor: Object not found
> > >
> > > #004: H5Gobj.c line 1047 in H5G_obj_lookup(): can't locate object
> > >
> > > major: Symbol table
> > > minor: Object not found
> > >
> > > #005: H5Gstab.c line 860 in H5G_stab_lookup(): not found
> > >
> > > major: Symbol table
> > > minor: Object not found
> > >
> > > #006: H5B.c line 360 in H5B_find(): can't lookup key in leaf node
> > >
> > > major: B-Tree node
> > > minor: Object not found
> > >
> > > #007: H5Gnode.c line 522 in H5G_node_found(): unable to protect symbol
> > > table
> > >
> > > node
> > >
> > > major: Symbol table
> > > minor: Unable to load metadata into cache
> > >
> > > #008: H5AC.c line 1831 in H5AC_protect(): H5C_protect() failed.
> > >
> > > major: Object cache
> > > minor: Unable to protect metadata
> > >
> > > #009: H5C.c line 6160 in H5C_protect(): can't load entry
> > >
> > > major: Object cache
> > > minor: Unable to load metadata into cache
> > >
> > > #010: H5C.c line 10990 in H5C_load_entry(): unable to load entry
> > >
> > > major: Object cache
> > > minor: Unable to load metadata into cache
> > >
> > > #011: H5Gcache.c line 176 in H5G_node_load(): bad symbol table node
> > >
> > > signature
> > >
> > > major: Symbol table
> > > minor: Unable to load metadata into cache
> > >
> > > I'm at a loss now, what am I doing wrong here?
> > >
> > > Cheers,
> > > Thorben
> > >
> > > _______________________________________________
> > > Hdf-forum is for HDF software users discussion.
> > > [email protected]
> > > http://**mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
[email protected] urgent: [email protected]
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
