You are confusing the size and extent of the datatype. The size (aka the physical number of bytes described by the memory layout) would be m*nloc*sizeof(type), while the extent will be related to where you expect the second element of the same type to start. If you do resize, you will incorporate the leading dimension in your pointer computation, and will see the gaps you were reporting.
George. On Thu, Dec 16, 2021 at 3:03 PM Jonas Thies via users < users@lists.open-mpi.org> wrote: > Dear Gilles, > > thanks, the resizing fixes the issue, it seems. It is not really > intuitive, though, because the actual extent of the data type is > m*nloc*sizeof(int) and I have to make MPI believe that it is > nloc*sizeof(int). And indeed, this seems to be not OpenMPI-specific, sorry > for that. > > Best, > > Jonas > > MPI_Type_vector (Gilles Gouaillardet) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 16 Dec 2021 10:29:27 +0100 > From: Jonas Thies <j.th...@tudelft.nl> <j.th...@tudelft.nl> > To: users@lists.open-mpi.org > Subject: [OMPI users] unexpected behavior when combining MPI_Gather > and MPI_Type_vector > Message-ID: <64075574-7a58-b194-208f-d455c10c8...@tudelft.nl> > <64075574-7a58-b194-208f-d455c10c8...@tudelft.nl> > Content-Type: text/plain; charset="utf-8"; Format="flowed" > > Dear OpenMPI community, > > Here's a little puzzle for the Christmas holidays (although I would > really appreciate a quick solution!). > > I'm stuck with the following relatively basic problem: given a local > nloc x m matrix X_p in column-major ordering on each MPI process p, > perform a single MPI_Gather operation to construct the matrix > X_0 > X_1 > ... > > X_nproc > > again, in col-major ordering. My approach is to use MPI_Type_vector to > define an stype and an rtype, where stype has stride nloc, and rtype has > stride nproc*nloc. The observation is that there is an unexpected > displacement of (m-1)*n*p in the result array for the part arriving from > process p. > > The MFE code is attached, and I use OpenMPI 4.0.5 with GCC 11.2 > (although other versions and even distributions seem to display the same > behavior). Example (nloc=3, nproc=3, m=2, with some additional columns > printed for the sake of demonstration): > > > > mpicxx -o matrix_gather matrix_gather.cpp > mpirun -np 3 ./matrix_gather > > v_loc on P0: 3x2 > 0 9 > 1 10 > 2 11 > > v_loc on P1: 3x2 > 3 12 > 4 13 > 5 14 > > v_loc on P2: 3x2 > 6 15 > 7 16 > 8 17 > > v_glob on P0: 9x4 > 0 9 0 0 > 1 10 0 0 > 2 11 0 0 > 0 3 12 0 > 0 4 13 0 > 0 5 14 0 > 0 0 6 15 > 0 0 7 16 > 0 0 8 17 > > Any ideas? > > Thanks, > > Jonas > > > > -- > *J. Thies* > Assistant Professor > > TU Delft > Faculty Electrical Engineering, Mathematics and Computer Science > Institute of Applied Mathematics and High Performance Computing Center > Mekelweg 4 > 2628 CD Delft > > T +31 15 27 XXXX > *j.th...@tudelft.nl <j.th...@tudelft.nl>* >
