Replacing Intel's build of MVAPICH2 2.2 with a fresh build of MVAPICH2 2.3b got me farther along. The comm mismatch does not seem to be a problem. I am guessing that the root cause was whatever bug is listed in http://mvapich.cse.ohio-state.edu/static/media/mvapich/MV2_CHANGELOG-2.3b.txt as:
- Fix hang in MPI_Probe - Thanks to John Westlund@Intel for the report I fixed the H5D__cmp_filtered_collective_io_info_entry_owner comparator, and now I'm back to fixing things about my patch to PETSc. I seem to be trying to filter a dataset that I shouldn't be. HDF5-DIAG: Error detected in HDF5 (1.11.0) MPI-process 0: #000: H5Dio.c line 319 in H5Dwrite(): can't prepare for writing data major: Dataset minor: Write failed #001: H5Dio.c line 395 in H5D__pre_write(): can't write data major: Dataset minor: Write failed #002: H5Dio.c line 831 in H5D__write(): unable to adjust I/O info for parallel I/O major: Dataset minor: Unable to initialize object #003: H5Dio.c line 1264 in H5D__ioinfo_adjust(): Can't perform independent write with filters in pipeline. The following caused a break from collective I/O: Local causes: Global causes: one of the dataspaces was neither simple nor scalar major: Low-level I/O minor: Can't perform independent IO On Wed, Nov 8, 2017 at 11:37 PM, Michael K. Edwards <m.k.edwa...@gmail.com> wrote: > Oddly enough, it is not the tag that is mismatched between receiver > and senders; it is io_info->comm. Something is decidedly out of whack > here. > > Rank 0, owner 0 probing with tag 0 on comm -1006632942 > Rank 2, owner 0 sent with tag 0 to comm -1006632952 as request 0 > Rank 3, owner 0 sent with tag 0 to comm -1006632952 as request 0 > Rank 1, owner 0 sent with tag 0 to comm -1006632952 as request 0 > > > On Wed, Nov 8, 2017 at 2:51 PM, Michael K. Edwards > <m.k.edwa...@gmail.com> wrote: >> >> I see that you're re-sorting by owner using a comparator called >> H5D__cmp_filtered_collective_io_info_entry_owner() which does not sort >> by a secondary key within items with equal owners. That, together >> with a sort that isn't stable (which HDqsort() probably isn't on most >> platforms; quicksort/introsort is not stable), will scramble the order >> in which different ranks traverse their local chunk arrays. That will >> cause deadly embraces between ranks that are waiting for each other's >> chunks to be sent. To fix that, it's probably sufficient to use the >> chunk offset as a secondary sort key in that comparator. >> >> That's not the root cause of the hang I'm currently experiencing, >> though. Still digging into that. >> >> >> On Wed, Nov 8, 2017 at 1:50 PM, Dana Robinson <derob...@hdfgroup.org> wrote: >> > Yes. All outside code that frees, allocates, or reallocates memory created >> > inside the library (or that will be passed back into the library, where it >> > could be freed or reallocated) should use these functions. This includes >> > filters. >> > >> > >> > >> > Dana >> > >> > >> > >> > From: Jordan Henderson <jhender...@hdfgroup.org> >> > Date: Wednesday, November 8, 2017 at 13:46 >> > To: Dana Robinson <derob...@hdfgroup.org>, "m.k.edwa...@gmail.com" >> > <m.k.edwa...@gmail.com>, HDF List <hdf-forum@lists.hdfgroup.org> >> > Subject: Re: [Hdf-forum] Collective IO and filters >> > >> > >> > >> > Dana, >> > >> > >> > >> > would it then make sense for all outside filters to use these routines? Due >> > to Parallel Compression's internal nature, it uses buffers allocated via >> > H5MM_ routines to collect and scatter data, which works fine for the >> > internal filters like deflate, since they use these as well. However, since >> > some of the outside filters use the raw malloc/free routines, causing >> > issues, I'm wondering if having all outside filters use the H5_ routines is >> > the cleanest solution.. >> > >> > >> > >> > Michael, >> > >> > >> > >> > Based on the "num_writers: 4" field, the NULL "receive_requests_array" and >> > the fact that for the same chunk, rank 0 shows "original owner: 0, new >> > owner: 0" and rank 3 shows "original owner: 3, new_owner: 0", it seems as >> > though everyone IS interested in the chunk the rank 0 is now working on, >> > but >> > now I'm more confident that at some point either the messages may have >> > failed to send or rank 0 is having problems finding the messages. >> > >> > >> > >> > Since in the unfiltered case it won't hit this particular code path, I'm >> > not >> > surprised that that case succeeds. If I had to make another guess based on >> > this, I would be inclined to think that rank 0 must be hanging on the >> > MPI_Mprobe due to a mismatch in the "tag" field. I use the index of the >> > chunk as the tag for the message in order to funnel specific messages to >> > the >> > correct rank for the correct chunk during the last part of the chunk >> > redistribution and if rank 0 can't match the tag it of course won't find >> > the >> > message. Why this might be happening, I'm not entirely certain currently. _______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5