Hi Quincey, I'm using H5T_NATIVE_DOUBLE to write an array of real*8 and H5T_NATIVE_REAL for real*4. Is that okay?
Kind regards, Stefan Frijters ________________________________________ From: [email protected] [[email protected]] On Behalf Of Quincey Koziol [[email protected]] Sent: 25 March 2010 22:40 To: HDF Users Discussion List Subject: Re: [Hdf-forum] HDF5 causes Fatal error in MPI_Gather Hi Stefan, On Mar 25, 2010, at 5:08 AM, Frijters, S.C.J. wrote: > Hi Quincey, > > I managed to increase the chunk size - I overlooked the fact that my blocks > of data weren't cubes in my testcase. However, it seems that performance can > suffer a lot for certain chunk sizes (for my test case): > > The size of my entire data array is 40 x 40 x 160. My MPI cartesian grid is 4 > x 4 x 1, so every core has a 10 x 10 x 160 subset. Originally I had the chunk > size set to 10 x 10 x 160 as well (which explains why I couldn't double the > 3rd component), and writes take less than a second. However, if I set the > chunk size to 20 x 20 x 160, it's really slow (7 seconds), while 40 x 40 x > 160 once again takes less than a second. I'd been reading up on the whole > chunking thing before, but I think I'm still ignorant of some of the > subtleties. Am I violating a rule here so that HDF goes back to independent > IO? Hmm, are your datatypes the same in memory and the file? If they aren't, HDF5 will break collective I/O down into independent I/O. > Is there some rule of thumb, or set of guidelines to get good performance out > of it? I read your "Parallel HDF5 Hints" document and some others, but it > hasn't helped me enough, apparently :-D. The time spent on IO in my > application is getting to be somewhat of a hot item. That would be where I'd point you. Quincey > Thanks again for the continued support, > > Stefan Frijters > ________________________________________ > From: [email protected] [[email protected]] On > Behalf Of Quincey Koziol [[email protected]] > Sent: 24 March 2010 17:12 > To: HDF Users Discussion List > Subject: Re: [Hdf-forum] HDF5 causes Fatal error in MPI_Gather > > Hi Stefan, > > On Mar 24, 2010, at 11:06 AM, Frijters, S.C.J. wrote: > >> Hi Quincey, >> >> I can double one dimension on my chunk size (at the cost of really slow IO), >> but if I double them all I get errors like these: >> >> HDF5-DIAG: Error detected in HDF5 (1.8.4) MPI-process 4: >> #000: H5D.c line 171 in H5Dcreate2(): unable to create dataset >> major: Dataset >> minor: Unable to initialize object >> #001: H5Dint.c line 428 in H5D_create_named(): unable to create and link to >> dataset >> major: Dataset >> minor: Unable to initialize object >> #002: H5L.c line 1639 in H5L_link_object(): unable to create new link to >> object >> major: Links >> minor: Unable to initialize object >> #003: H5L.c line 1862 in H5L_create_real(): can't insert link >> major: Symbol table >> minor: Unable to insert object >> #004: H5Gtraverse.c line 877 in H5G_traverse(): internal path traversal >> failed >> major: Symbol table >> minor: Object not found >> #005: H5Gtraverse.c line 703 in H5G_traverse_real(): traversal operator >> failed >> major: Symbol table >> minor: Callback failed >> #006: H5L.c line 1685 in H5L_link_cb(): unable to create object >> major: Object header >> minor: Unable to initialize object >> #007: H5O.c line 2677 in H5O_obj_create(): unable to open object >> major: Object header >> minor: Can't open object >> #008: H5Doh.c line 296 in H5O_dset_create(): unable to create dataset >> major: Dataset >> minor: Unable to initialize object >> #009: H5Dint.c line 1030 in H5D_create(): unable to construct layout >> information >> major: Dataset >> minor: Unable to initialize object >> #010: H5Dchunk.c line 420 in H5D_chunk_construct(): chunk size must be <= >> maximum dimension size for fixed-sized dimensions >> major: Dataset >> minor: Unable to initialize object >> >> I am currently doing test runs on my local machine on 16 cores because the >> large machine I run jobs on is unavailable at the moment and has a queueing >> system rather unsuited to quick test runs, so maybe this is an artefact of >> running on such a small number of cores? Although I *think* I tried this >> before and got the same type of error on several thousand cores also. > > You seem to have increased the chunk dimension to be larger than the > dataset dimension. What is the chunk size and dataspace size you are using? > > Quincey > >> >> Kind regards, >> >> Stefan Frijters >> >> ________________________________________ >> From: [email protected] [[email protected]] On >> Behalf Of Quincey Koziol [[email protected]] >> Sent: 24 March 2010 16:28 >> To: HDF Users Discussion List >> Subject: Re: [Hdf-forum] HDF5 causes Fatal error in MPI_Gather >> >> Hi Stefan, >> >> On Mar 24, 2010, at 10:10 AM, Stefan Frijters wrote: >> >>> Hi Quincey, >>> >>> Thanks for the quick response. Currently, each core is handling its >>> datasets with a chunk size equal to the size of the local data (the dims >>> parameter in h5pset_chunk_f is equal to to dims parameter in >>> h5dwrite_f) because the local arrays are not that large anyway (in the >>> order of 20x20x20 reals), so if I understand things correctly I'm >>> already using maximum chunk size. >> >> No, you don't have to make them the same size, since the collective >> I/O should stitch them back together anyway. Try doubling the dimensions on >> your chunks. >> >>> Do you have an idea why it doesn't crash the first time I try to do it >>> though? It's a different array, but of the same size and datatype as the >>> second. As far as I can see I'm closing all used handles at the end of >>> my function at least. >> >> Hmm, I'm not certain... >> >> Quincey >> >>> Kind regards, >>> >>> Stefan Frijters >>> >>>> Hi Stefan, >>>> >>>> On Mar 24, 2010, at 3:11 AM, Stefan Frijters wrote: >>>> >>>>> Dear all, >>>>> >>>>> Recently, I've run into a problem with my parallel HDF5 writes. My >>>>> program works fine on 8k cores, but when I run it on 16k cores it >>>>> crashes when writing a data file through h5dwrite_f(...). >>>>> File writes go through one function in the code, so it always uses the >>>>> same code, but for some reason I don't understand it writes one file >>>>> without problems, but the second one throws the following error message: >>>>> >>>>> Abort(1) on node 0 (rank 0 in comm 1140850688): Fatal error in >>>>> MPI_Gather: Invalid buffer pointer, error stack: >>>>> MPI_Gather(758): MPI_Gather(sbuf=0xa356f400, scount=16000, MPI_BYTE, >>>>> rbuf=(nil), rcount=16000, MPI_BYTE, root=0, comm=0x84000003) failed >>>>> MPI_Gather(675): Null buffer pointer >>>>> >>>>> I've been looking through the HDF5 source code and it only seems to call >>>>> MPI_Gather in one place, in the function H5D_obtain_mpio_mode. In that >>>>> function HDF tries to allocate a receive buffer using >>>>> >>>>> recv_io_mode_info = (uint8_t *)H5MM_malloc(total_chunks * mpi_size); >>>>> >>>>> Which then returns the null pointer seen in rbuf=(nil) instead of a >>>>> valid pointer. Thus, to me it seems it's HDF causing the problem and not >>>>> MPI. >>>>> >>>>> This problem occurs in both collective and independent IO mode. >>>>> >>>>> Do you have any idea what might be causing this problem, or how to >>>>> resolve it? I'm not sure what kind of other information you might need, >>>>> but I'll do my best to supply it, if you need any. >>>> >>>> This is a scalability problem we are aware of and are working to address, >>>> but in the meanwhile, can you increase the size of your chunks for your >>>> dataset(s)? (That will reduce the number of chunks and the size of the >>>> buffer being allocated) >>>> >>>> Quincey >>>> >>> >>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
