Re: [OMPI users] unexpected behavior when combining MPI_Gather and MPI_Type_vector
Jonas, The section 5.1.6 in MPI 4.0 should give you a better idea about the differences between size, extent and true extent. There are also few examples in Section 5.1.14 on how to manipulate the datatype using extent. I think you should find Examples 5.13 to 5.16 of particular interest. Best, George. On Thu, Dec 16, 2021 at 5:15 PM Jonas Thies wrote: > Hi George, > > thanks, I'm starting to understand this now. > > Still not quite intuitive that "Type_create_resized" allows me to reset > the extent but not the size (just from a naming perspective). > > The man page is talking about extent, upper and lower bounds, but the > upper bound cannot be specified: > > NAME >MPI_Type_create_resized - Returns a new data type with new extent > and >upper and lower bounds. > > SYNTAX > C Syntax >#include >int MPI_Type_create_resized(MPI_Datatype oldtype, MPI_Aint lb, > MPI_Aint extent, MPI_Datatype *newtype) > > > Jonas > On 16-12-2021 22:39, George Bosilca wrote: > > You are confusing the size and extent of the datatype. The size (aka the > physical number of bytes described by the memory layout) would be > m*nloc*sizeof(type), while the extent will be related to where you expect > the second element of the same type to start. If you do resize, you will > incorporate the leading dimension in your pointer computation, and will see > the gaps you were reporting. > > George. > > > > > On Thu, Dec 16, 2021 at 3:03 PM Jonas Thies via users < > users@lists.open-mpi.org> wrote: > >> Dear Gilles, >> >> thanks, the resizing fixes the issue, it seems. It is not really >> intuitive, though, because the actual extent of the data type is >> m*nloc*sizeof(int) and I have to make MPI believe that it is >> nloc*sizeof(int). And indeed, this seems to be not OpenMPI-specific, sorry >> for that. >> >> Best, >> >> Jonas >> >> MPI_Type_vector (Gilles Gouaillardet) >> >> >> ---------- >> >> Message: 1 >> Date: Thu, 16 Dec 2021 10:29:27 +0100 >> From: Jonas Thies >> To: users@lists.open-mpi.org >> Subject: [OMPI users] unexpected behavior when combining MPI_Gather >> and MPI_Type_vector >> Message-ID: <64075574-7a58-b194-208f-d455c10c8...@tudelft.nl> >> <64075574-7a58-b194-208f-d455c10c8...@tudelft.nl> >> Content-Type: text/plain; charset="utf-8"; Format="flowed" >> >> Dear OpenMPI community, >> >> Here's a little puzzle for the Christmas holidays (although I would >> really appreciate a quick solution!). >> >> I'm stuck with the following relatively basic problem: given a local >> nloc x m matrix X_p in column-major ordering on each MPI process p, >> perform a single MPI_Gather operation to construct the matrix >> X_0 >> X_1 >> ... >> >> X_nproc >> >> again, in col-major ordering. My approach is to use MPI_Type_vector to >> define an stype and an rtype, where stype has stride nloc, and rtype has >> stride nproc*nloc. The observation is that there is an unexpected >> displacement of (m-1)*n*p in the result array for the part arriving from >> process p. >> >> The MFE code is attached, and I use OpenMPI 4.0.5 with GCC 11.2 >> (although other versions and even distributions seem to display the same >> behavior). Example (nloc=3, nproc=3, m=2, with some additional columns >> printed for the sake of demonstration): >> >> >> > mpicxx -o matrix_gather matrix_gather.cpp >> mpirun -np 3 ./matrix_gather >> >> v_loc on P0: 3x2 >> 0 9 >> 1 10 >> 2 11 >> >> v_loc on P1: 3x2 >> 3 12 >> 4 13 >> 5 14 >> >> v_loc on P2: 3x2 >> 6 15 >> 7 16 >> 8 17 >> >> v_glob on P0: 9x4 >> 0 9 0 0 >> 1 10 0 0 >> 2 11 0 0 >> 0 3 12 0 >> 0 4 13 0 >> 0 5 14 0 >> 0 0 6 15 >> 0 0 7 16 >> 0 0 8 17 >> >> Any ideas? >> >> Thanks, >> >> Jonas >> >> >> >> -- >> *J. Thies* >> Assistant Professor >> >> TU Delft >> Faculty Electrical Engineering, Mathematics and Computer Science >> Institute of Applied Mathematics and High Performance Computing Center >> Mekelweg 4 >> 2628 CD Delft >> >> T +31 15 27 >> *j.th...@tudelft.nl * >> > -- > *J. Thies* > Assistant Professor > > TU Delft > Faculty Electrical Engineering, Mathematics and Computer Science > Institute of Applied Mathematics and High Performance Computing Center > Mekelweg 4 > 2628 CD Delft > > T +31 15 27 > *j.th...@tudelft.nl * >
Re: [OMPI users] unexpected behavior when combining MPI_Gather and MPI_Type_vector
Hi George, thanks, I'm starting to understand this now. Still not quite intuitive that "Type_create_resized" allows me to reset the extent but not the size (just from a naming perspective). The man page is talking about extent, upper and lower bounds, but the upper bound cannot be specified: NAME MPI_Type_create_resized - Returns a new data type with new extent and upper and lower bounds. SYNTAX C Syntax #include int MPI_Type_create_resized(MPI_Datatype oldtype, MPI_Aint lb, MPI_Aint extent, MPI_Datatype *newtype) Jonas On 16-12-2021 22:39, George Bosilca wrote: You are confusing the size and extent of the datatype. The size (aka the physical number of bytes described by the memory layout) would be m*nloc*sizeof(type), while the extent will be related to where you expect the second element of the same type to start. If you do resize, you will incorporate the leading dimension in your pointer computation, and will see the gaps you were reporting. George. On Thu, Dec 16, 2021 at 3:03 PM Jonas Thies via users mailto:users@lists.open-mpi.org>> wrote: Dear Gilles, thanks, the resizing fixes the issue, it seems. It is not really intuitive, though, because the actual extent of the data type is m*nloc*sizeof(int) and I have to make MPI believe that it is nloc*sizeof(int). And indeed, this seems to be not OpenMPI-specific, sorry for that. Best, Jonas MPI_Type_vector (Gilles Gouaillardet) -- Message: 1 Date: Thu, 16 Dec 2021 10:29:27 +0100 From: Jonas Thies <mailto:j.th...@tudelft.nl> To:users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> Subject: [OMPI users] unexpected behavior when combining MPI_Gather and MPI_Type_vector Message-ID:<64075574-7a58-b194-208f-d455c10c8...@tudelft.nl> <mailto:64075574-7a58-b194-208f-d455c10c8...@tudelft.nl> Content-Type: text/plain; charset="utf-8"; Format="flowed" Dear OpenMPI community, Here's a little puzzle for the Christmas holidays (although I would really appreciate a quick solution!). I'm stuck with the following relatively basic problem: given a local nloc x m matrix X_p in column-major ordering on each MPI process p, perform a single MPI_Gather operation to construct the matrix X_0 X_1 ... X_nproc again, in col-major ordering. My approach is to use MPI_Type_vector to define an stype and an rtype, where stype has stride nloc, and rtype has stride nproc*nloc. The observation is that there is an unexpected displacement of (m-1)*n*p in the result array for the part arriving from process p. The MFE code is attached, and I use OpenMPI 4.0.5 with GCC 11.2 (although other versions and even distributions seem to display the same behavior). Example (nloc=3, nproc=3, m=2, with some additional columns printed for the sake of demonstration): > mpicxx -o matrix_gather matrix_gather.cpp mpirun -np 3 ./matrix_gather v_loc on P0: 3x2 0 9 1 10 2 11 v_loc on P1: 3x2 3 12 4 13 5 14 v_loc on P2: 3x2 6 15 7 16 8 17 v_glob on P0: 9x4 0 9 0 0 1 10 0 0 2 11 0 0 0 3 12 0 0 4 13 0 0 5 14 0 0 0 6 15 0 0 7 16 0 0 8 17 Any ideas? Thanks, Jonas -- *J. Thies* Assistant Professor TU Delft Faculty Electrical Engineering, Mathematics and Computer Science Institute of Applied Mathematics and High Performance Computing Center Mekelweg 4 2628 CD Delft T +31 15 27 *j.th...@tudelft.nl <mailto:j.th...@tudelft.nl>* -- *J. Thies* Assistant Professor TU Delft Faculty Electrical Engineering, Mathematics and Computer Science Institute of Applied Mathematics and High Performance Computing Center Mekelweg 4 2628 CD Delft T +31 15 27 *j.th...@tudelft.nl*
Re: [OMPI users] unexpected behavior when combining MPI_Gather and MPI_Type_vector
You are confusing the size and extent of the datatype. The size (aka the physical number of bytes described by the memory layout) would be m*nloc*sizeof(type), while the extent will be related to where you expect the second element of the same type to start. If you do resize, you will incorporate the leading dimension in your pointer computation, and will see the gaps you were reporting. George. On Thu, Dec 16, 2021 at 3:03 PM Jonas Thies via users < users@lists.open-mpi.org> wrote: > Dear Gilles, > > thanks, the resizing fixes the issue, it seems. It is not really > intuitive, though, because the actual extent of the data type is > m*nloc*sizeof(int) and I have to make MPI believe that it is > nloc*sizeof(int). And indeed, this seems to be not OpenMPI-specific, sorry > for that. > > Best, > > Jonas > > MPI_Type_vector (Gilles Gouaillardet) > > > -- > > Message: 1 > Date: Thu, 16 Dec 2021 10:29:27 +0100 > From: Jonas Thies > To: users@lists.open-mpi.org > Subject: [OMPI users] unexpected behavior when combining MPI_Gather > and MPI_Type_vector > Message-ID: <64075574-7a58-b194-208f-d455c10c8...@tudelft.nl> > <64075574-7a58-b194-208f-d455c10c8...@tudelft.nl> > Content-Type: text/plain; charset="utf-8"; Format="flowed" > > Dear OpenMPI community, > > Here's a little puzzle for the Christmas holidays (although I would > really appreciate a quick solution!). > > I'm stuck with the following relatively basic problem: given a local > nloc x m matrix X_p in column-major ordering on each MPI process p, > perform a single MPI_Gather operation to construct the matrix > X_0 > X_1 > ... > > X_nproc > > again, in col-major ordering. My approach is to use MPI_Type_vector to > define an stype and an rtype, where stype has stride nloc, and rtype has > stride nproc*nloc. The observation is that there is an unexpected > displacement of (m-1)*n*p in the result array for the part arriving from > process p. > > The MFE code is attached, and I use OpenMPI 4.0.5 with GCC 11.2 > (although other versions and even distributions seem to display the same > behavior). Example (nloc=3, nproc=3, m=2, with some additional columns > printed for the sake of demonstration): > > > > mpicxx -o matrix_gather matrix_gather.cpp > mpirun -np 3 ./matrix_gather > > v_loc on P0: 3x2 > 0 9 > 1 10 > 2 11 > > v_loc on P1: 3x2 > 3 12 > 4 13 > 5 14 > > v_loc on P2: 3x2 > 6 15 > 7 16 > 8 17 > > v_glob on P0: 9x4 > 0 9 0 0 > 1 10 0 0 > 2 11 0 0 > 0 3 12 0 > 0 4 13 0 > 0 5 14 0 > 0 0 6 15 > 0 0 7 16 > 0 0 8 17 > > Any ideas? > > Thanks, > > Jonas > > > > -- > *J. Thies* > Assistant Professor > > TU Delft > Faculty Electrical Engineering, Mathematics and Computer Science > Institute of Applied Mathematics and High Performance Computing Center > Mekelweg 4 > 2628 CD Delft > > T +31 15 27 > *j.th...@tudelft.nl * >
Re: [OMPI users] unexpected behavior when combining MPI_Gather and MPI_Type_vector
Dear Gilles, thanks, the resizing fixes the issue, it seems. It is not really intuitive, though, because the actual extent of the data type is m*nloc*sizeof(int) and I have to make MPI believe that it is nloc*sizeof(int). And indeed, this seems to be not OpenMPI-specific, sorry for that. Best, Jonas MPI_Type_vector (Gilles Gouaillardet) -- Message: 1 Date: Thu, 16 Dec 2021 10:29:27 +0100 From: Jonas Thies To: users@lists.open-mpi.org Subject: [OMPI users] unexpected behavior when combining MPI_Gather and MPI_Type_vector Message-ID: <64075574-7a58-b194-208f-d455c10c8...@tudelft.nl> Content-Type: text/plain; charset="utf-8"; Format="flowed" Dear OpenMPI community, Here's a little puzzle for the Christmas holidays (although I would really appreciate a quick solution!). I'm stuck with the following relatively basic problem: given a local nloc x m matrix X_p in column-major ordering on each MPI process p, perform a single MPI_Gather operation to construct the matrix X_0 X_1 ... X_nproc again, in col-major ordering. My approach is to use MPI_Type_vector to define an stype and an rtype, where stype has stride nloc, and rtype has stride nproc*nloc. The observation is that there is an unexpected displacement of (m-1)*n*p in the result array for the part arriving from process p. The MFE code is attached, and I use OpenMPI 4.0.5 with GCC 11.2 (although other versions and even distributions seem to display the same behavior). Example (nloc=3, nproc=3, m=2, with some additional columns printed for the sake of demonstration): > mpicxx -o matrix_gather matrix_gather.cpp mpirun -np 3 ./matrix_gather v_loc on P0: 3x2 0 9 1 10 2 11 v_loc on P1: 3x2 3 12 4 13 5 14 v_loc on P2: 3x2 6 15 7 16 8 17 v_glob on P0: 9x4 0 9 0 0 1 10 0 0 2 11 0 0 0 3 12 0 0 4 13 0 0 5 14 0 0 0 6 15 0 0 7 16 0 0 8 17 Any ideas? Thanks, Jonas -- *J. Thies* Assistant Professor TU Delft Faculty Electrical Engineering, Mathematics and Computer Science Institute of Applied Mathematics and High Performance Computing Center Mekelweg 4 2628 CD Delft T +31 15 27 *j.th...@tudelft.nl*
Re: [OMPI users] unexpected behavior when combining MPI_Gather and MPI_Type_vector
Jonas,
In case I misunderstood your question and you want to print
v_glob on P0: 9x2
0 9
1 10
2 11
3 12
4 13
5 14
6 15
7 16
8 17
then you have to fix the print invocation
// note: print an additional column to show the displacement error we get:
if (!rank) print("v_glob", rank, n, m, v_glob);
and also resize rtype so the second element starts at v_glob[3][0] => upper
bound = (3*sizeof(int))
By the way, since this question is not Open MPI specific, sites such as
Stack Overflow are a better fit.
Cheers,
Gilles
On Thu, Dec 16, 2021 at 6:46 PM Gilles Gouaillardet <
gilles.gouaillar...@gmail.com> wrote:
> Jonas,
>
> Assuming v_glob is what you expect, you will need to
> `MPI_Type_create_resized_type()` the received type so the block received
> from process 1 will be placed at the right position (v_glob[3][1] => upper
> bound = ((4*3+1) * sizeof(int))
>
> Cheers,
>
> Gilles
>
> On Thu, Dec 16, 2021 at 6:33 PM Jonas Thies via users <
> users@lists.open-mpi.org> wrote:
>
>> Dear OpenMPI community,
>>
>> Here's a little puzzle for the Christmas holidays (although I would
>> really appreciate a quick solution!).
>>
>> I'm stuck with the following relatively basic problem: given a local nloc
>> x m matrix X_p in column-major ordering on each MPI process p, perform a
>> single MPI_Gather operation to construct the matrix
>> X_0
>> X_1
>> ...
>>
>> X_nproc
>>
>> again, in col-major ordering. My approach is to use MPI_Type_vector to
>> define an stype and an rtype, where stype has stride nloc, and rtype has
>> stride nproc*nloc. The observation is that there is an unexpected
>> displacement of (m-1)*n*p in the result array for the part arriving from
>> process p.
>>
>> The MFE code is attached, and I use OpenMPI 4.0.5 with GCC 11.2 (although
>> other versions and even distributions seem to display the same behavior).
>> Example (nloc=3, nproc=3, m=2, with some additional columns printed for the
>> sake of demonstration):
>>
>>
>> > mpicxx -o matrix_gather matrix_gather.cpp
>> mpirun -np 3 ./matrix_gather
>>
>> v_loc on P0: 3x2
>> 0 9
>> 1 10
>> 2 11
>>
>> v_loc on P1: 3x2
>> 3 12
>> 4 13
>> 5 14
>>
>> v_loc on P2: 3x2
>> 6 15
>> 7 16
>> 8 17
>>
>> v_glob on P0: 9x4
>> 0 9 0 0
>> 1 10 0 0
>> 2 11 0 0
>> 0 3 12 0
>> 0 4 13 0
>> 0 5 14 0
>> 0 0 6 15
>> 0 0 7 16
>> 0 0 8 17
>>
>> Any ideas?
>>
>> Thanks,
>>
>> Jonas
>>
>>
>> --
>> *J. Thies*
>> Assistant Professor
>>
>> TU Delft
>> Faculty Electrical Engineering, Mathematics and Computer Science
>> Institute of Applied Mathematics and High Performance Computing Center
>> Mekelweg 4
>> 2628 CD Delft
>>
>> T +31 15 27
>> *j.th...@tudelft.nl *
>>
>
Re: [OMPI users] unexpected behavior when combining MPI_Gather and MPI_Type_vector
Jonas, Assuming v_glob is what you expect, you will need to `MPI_Type_create_resized_type()` the received type so the block received from process 1 will be placed at the right position (v_glob[3][1] => upper bound = ((4*3+1) * sizeof(int)) Cheers, Gilles On Thu, Dec 16, 2021 at 6:33 PM Jonas Thies via users < users@lists.open-mpi.org> wrote: > Dear OpenMPI community, > > Here's a little puzzle for the Christmas holidays (although I would really > appreciate a quick solution!). > > I'm stuck with the following relatively basic problem: given a local nloc > x m matrix X_p in column-major ordering on each MPI process p, perform a > single MPI_Gather operation to construct the matrix > X_0 > X_1 > ... > > X_nproc > > again, in col-major ordering. My approach is to use MPI_Type_vector to > define an stype and an rtype, where stype has stride nloc, and rtype has > stride nproc*nloc. The observation is that there is an unexpected > displacement of (m-1)*n*p in the result array for the part arriving from > process p. > > The MFE code is attached, and I use OpenMPI 4.0.5 with GCC 11.2 (although > other versions and even distributions seem to display the same behavior). > Example (nloc=3, nproc=3, m=2, with some additional columns printed for the > sake of demonstration): > > > > mpicxx -o matrix_gather matrix_gather.cpp > mpirun -np 3 ./matrix_gather > > v_loc on P0: 3x2 > 0 9 > 1 10 > 2 11 > > v_loc on P1: 3x2 > 3 12 > 4 13 > 5 14 > > v_loc on P2: 3x2 > 6 15 > 7 16 > 8 17 > > v_glob on P0: 9x4 > 0 9 0 0 > 1 10 0 0 > 2 11 0 0 > 0 3 12 0 > 0 4 13 0 > 0 5 14 0 > 0 0 6 15 > 0 0 7 16 > 0 0 8 17 > > Any ideas? > > Thanks, > > Jonas > > > -- > *J. Thies* > Assistant Professor > > TU Delft > Faculty Electrical Engineering, Mathematics and Computer Science > Institute of Applied Mathematics and High Performance Computing Center > Mekelweg 4 > 2628 CD Delft > > T +31 15 27 > *j.th...@tudelft.nl * >
[OMPI users] unexpected behavior when combining MPI_Gather and MPI_Type_vector
Dear OpenMPI community,
Here's a little puzzle for the Christmas holidays (although I would
really appreciate a quick solution!).
I'm stuck with the following relatively basic problem: given a local
nloc x m matrix X_p in column-major ordering on each MPI process p,
perform a single MPI_Gather operation to construct the matrix
X_0
X_1
...
X_nproc
again, in col-major ordering. My approach is to use MPI_Type_vector to
define an stype and an rtype, where stype has stride nloc, and rtype has
stride nproc*nloc. The observation is that there is an unexpected
displacement of (m-1)*n*p in the result array for the part arriving from
process p.
The MFE code is attached, and I use OpenMPI 4.0.5 with GCC 11.2
(although other versions and even distributions seem to display the same
behavior). Example (nloc=3, nproc=3, m=2, with some additional columns
printed for the sake of demonstration):
> mpicxx -o matrix_gather matrix_gather.cpp
mpirun -np 3 ./matrix_gather
v_loc on P0: 3x2
0 9
1 10
2 11
v_loc on P1: 3x2
3 12
4 13
5 14
v_loc on P2: 3x2
6 15
7 16
8 17
v_glob on P0: 9x4
0 9 0 0
1 10 0 0
2 11 0 0
0 3 12 0
0 4 13 0
0 5 14 0
0 0 6 15
0 0 7 16
0 0 8 17
Any ideas?
Thanks,
Jonas
--
*J. Thies*
Assistant Professor
TU Delft
Faculty Electrical Engineering, Mathematics and Computer Science
Institute of Applied Mathematics and High Performance Computing Center
Mekelweg 4
2628 CD Delft
T +31 15 27
*j.th...@tudelft.nl*
#include
#include
#include
#include
void print(std::string label, int rank, int nloc, int m, int* array)
{
std::ostringstream oss;
oss << label << " on P"<
