Re: [OMPI users] MPI_Alltoall with Vector Datatype

[Spenser Gilliland]

George,

I figured it out.  The defined type was

MPI_Type_vector(N, wrows, N, MPI_FLOAT, _all_unaligned_t);

Where it should have been

MPI_Type_vector(wrows, wrows, N, MPI_FLOAT, _all_unaligned_t);

This clears up all the errors.

Thanks,
Spenser

On Thu, May 8, 2014 at 5:43 PM, Spenser Gilliland
 wrote:
> George,
>
>> The alltoall exchanges data from all nodes to all nodes, including the
>> local participant. So every participant will write the same amount of
>> data.
>
> Yes, I believe that is what my code is doing.  However, I'm not sure
> why the out of bounds is occurring.  Can you be more specific?  I
> really want to get this working.
>
> Thanks,
> Spenser
>
>
>
> --
> Spenser Gilliland
> Computer Engineer
> Doctoral Candidate



-- 
Spenser Gilliland
Computer Engineer
Doctoral Candidate

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[Spenser Gilliland]

George,

> The alltoall exchanges data from all nodes to all nodes, including the
> local participant. So every participant will write the same amount of
> data.

Yes, I believe that is what my code is doing.  However, I'm not sure
why the out of bounds is occurring.  Can you be more specific?  I
really want to get this working.

Thanks,
Spenser



-- 
Spenser Gilliland
Computer Engineer
Doctoral Candidate

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[George Bosilca]

The alltoall exchanges data from all nodes to all nodes, including the
local participant. So every participant will write the same amount of
data.

  George.


On Thu, May 8, 2014 at 6:16 PM, Spenser Gilliland
 wrote:
> George,
>
>> Here is basically what is happening. On the top left, I depicted the 
>> datatype resulting from the vector type. The two arrows point to the lower 
>> bound and upper bound (thus the extent) of the datatype. On the top right, 
>> the resized datatype, where the ub is now moved 2 elements after the lb, 
>> allowing for a nice interleaving of the data. Then the next line is the 
>> unrolled datatype representation, flatten to a 1D. Again it contains in red 
>> the data touched by the defined memory layout, as well as the extent (lb and 
>> ub).
>>
>> Now, let’s move on the MPI_Alltoall call. The array is the one without 
>> colors, and then I put the datatype starting from the position you specified 
>> in the alltoall. As you can see as soon as you don’t start at the origin of 
>> the allocated memory, you end-up writing outside of your data. This happens 
>> deep inside the MPI_Alltoall call (no validation at the MPI level).
>
> Why are the last two elements in the 1D view present?  If that's the
> case I would have to define a new MPI Type for each set of columns
> within a matrix.  Why would it be defined in this manner?  Also, why
> is the extent of the initial vector type equal to 12 when it is
> actually accessing 16 elements (for the 4x4 example).
>
> So, is this a bug in Alltoall or openmpi?
>
> I believe it is all to all causing the bug and not vector because the 
> following
>
> MPI_Aint lb, extent, true_lb, true_extent;
> MPI_Type_get_extent(mpi_all_t, , );
> MPI_Type_get_true_extent(mpi_all_t, _lb, _extent);
> printf("mpi_all_t - lb = %d, extent = %d, true_lb = %d, true_extent =
> %d\n", lb, extent, true_lb, true_extent);
>
> produces
>
> mpi_all_t - lb = 0, extent = 16, true_lb = 0, true_extent = 240
>
> Which means that the size is correct (using 4 byte floats with 2
> processor on an 8x8 array this would be the 30th element).
>
> There's a similar drawing to what you made attached that's more
> focused on the specific instance in this code.  Hopefully, this clears
> up the algorithm a bit.
>
> Thanks,
> Spenser
>
> --
> Spenser Gilliland
> Computer Engineer
> Doctoral Candidate
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[Spenser Gilliland]

George,

> Here is basically what is happening. On the top left, I depicted the datatype 
> resulting from the vector type. The two arrows point to the lower bound and 
> upper bound (thus the extent) of the datatype. On the top right, the resized 
> datatype, where the ub is now moved 2 elements after the lb, allowing for a 
> nice interleaving of the data. Then the next line is the unrolled datatype 
> representation, flatten to a 1D. Again it contains in red the data touched by 
> the defined memory layout, as well as the extent (lb and ub).
>
> Now, let’s move on the MPI_Alltoall call. The array is the one without 
> colors, and then I put the datatype starting from the position you specified 
> in the alltoall. As you can see as soon as you don’t start at the origin of 
> the allocated memory, you end-up writing outside of your data. This happens 
> deep inside the MPI_Alltoall call (no validation at the MPI level).

Why are the last two elements in the 1D view present?  If that's the
case I would have to define a new MPI Type for each set of columns
within a matrix.  Why would it be defined in this manner?  Also, why
is the extent of the initial vector type equal to 12 when it is
actually accessing 16 elements (for the 4x4 example).

So, is this a bug in Alltoall or openmpi?

I believe it is all to all causing the bug and not vector because the following

MPI_Aint lb, extent, true_lb, true_extent;
MPI_Type_get_extent(mpi_all_t, , );
MPI_Type_get_true_extent(mpi_all_t, _lb, _extent);
printf("mpi_all_t - lb = %d, extent = %d, true_lb = %d, true_extent =
%d\n", lb, extent, true_lb, true_extent);

produces

mpi_all_t - lb = 0, extent = 16, true_lb = 0, true_extent = 240

Which means that the size is correct (using 4 byte floats with 2
processor on an 8x8 array this would be the 30th element).

There's a similar drawing to what you made attached that's more
focused on the specific instance in this code.  Hopefully, this clears
up the algorithm a bit.

Thanks,
Spenser

-- 
Spenser Gilliland
Computer Engineer
Doctoral Candidate


MPI_Alltoall_Derived_Types.pdf
Description: Adobe PDF document

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[George Bosilca]

Spenser,

Here is basically what is happening. On the top left, I depicted the datatype 
resulting from the vector type. The two arrows point to the lower bound and 
upper bound (thus the extent) of the datatype. On the top right, the resized 
datatype, where the ub is now moved 2 elements after the lb, allowing for a 
nice interleaving of the data. Then the next line is the unrolled datatype 
representation, flatten to a 1D. Again it contains in red the data touched by 
the defined memory layout, as well as the extent (lb and ub).

Now, let’s move on the MPI_Alltoall call. The array is the one without colors, 
and then I put the datatype starting from the position you specified in the 
alltoall. As you can see as soon as you don’t start at the origin of the 
allocated memory, you end-up writing outside of your data. This happens deep 
inside the MPI_Alltoall call (no validation at the MPI level).

Hope this helps,
  George.



alltoall_with_datatype.pdf
Description: Adobe PDF document

On May 8, 2014, at 16:02 , Spenser Gilliland  wrote:

> Matthieu & George,
> 
> Thanks you both for helping me. I really appreciate it.
> 
>> A simple test would be to run it with valgrind, so that out of bound
>> reads and writes will be obvious.
> 
> I ran it through valgrind (i left the command line I used in there so
> you can verify the methods)
> 
> I am getting errors with valgrind inside the Alltoall function.  See
> https://gist.github.com/anonymous/fbd83343f456f0688cea .
> 
> These errors do not occur in the stack allocated version.  See
> https://gist.github.com/anonymous/f4dbcddbbc9fee0f508e .  I assume
> this is due to valgrind being unable to detect stack corruption.
> 
>> The segfault indicates that you overwrite outside of the allocated memory 
>> (and conflicts with the ptmalloc
>> library). I’m quite certain that you write outside the allocated array …
> 
> So, my understanding is that Alltoall would utilize wsize multiplied
> by the size of the data type indexes of the matrix and jump extent
> bytes between communications.  Thus, I'm very lost as to why the
> Alltoall is exceeding the bounds of my array.
> 
> Thanks,
> Spenser
> 
> On Thu, May 8, 2014 at 2:19 PM, Matthieu Brucher
>  wrote:
>> A simple test would be to run it with valgrind, so that out of bound
>> reads and writes will be obvious.
>> 
>> Cheers,
>> 
>> Matthieu
>> 
>> 2014-05-08 21:16 GMT+02:00 Spenser Gilliland :
>>> George & Mattheiu,
>>> 
 The Alltoall should only return when all data is sent and received on
 the current rank, so there shouldn't be any race condition.
>>> 
>>> Your right this is MPI not pthreads.  That should never happen. Duh!
>>> 
 I think the issue is with the way you define the send and receive
 buffer in the MPI_Alltoall. You have to keep in mind that the
 all-to-all pattern will overwrite the entire data in the receive
 buffer. Thus, starting from a relative displacement in the data (in
 this case matrix[wrank*wrows]), begs for troubles, as you will write
 outside the receive buffer.
>>> 
>>> The submatrix corresponding to matrix[wrank*wrows][0] to
>>> matrix[(wrank+1)*wrows-1][:] is valid only on the wrank process.  This
>>> is a block distribution of the rows like what MPI_Scatter would
>>> produce.  As wrows is equal to N (matrix width/height) divided by
>>> wsize, the number of mpi_all_t blocks in each message is equal to
>>> wsize.  Therefore, there should be no writing outside the bounds of
>>> the submatrix.
>>> 
>>> On another note,
>>> I just ported the example to use dynamic memory and now I'm getting
>>> segfaults when I call MPI_Finalize().  Any idea what in the code could
>>> have caused this?
>>> 
>>> It's paste binned here: 
>>> https://gist.github.com/anonymous/a80e0679c3cbffb82e39
>>> 
>>> The result is
>>> 
>>> [sgillila@jarvis src]$ mpirun -npernode 2 transpose2 8
>>> N = 8
>>> Matrix =
>>> 0: 0 1 2 3 4 5 6 7
>>> 0: 8 9101112131415
>>> 0:1617181920212223
>>> 0:2425262728293031
>>> 1:3233343536373839
>>> 1:4041424344454647
>>> 1:4849505152535455
>>> 1:5657585960616263
>>> Matrix =
>>> 0: 0 8162432404856
>>> 0: 1 9172533414957
>>> 0: 210182634425058
>>> 0: 311192735435159
>>> 1: 412202836445260
>>> 1: 513212937455361
>>> 1: 614223038465462
>>> 1: 715233139475563
>>> [jarvis:09314] *** Process received signal ***
>>> [jarvis:09314] Signal: Segmentation fault (11)
>>> [jarvis:09314] 

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[Spenser Gilliland]

Matthieu & George,

Thanks you both for helping me. I really appreciate it.

> A simple test would be to run it with valgrind, so that out of bound
> reads and writes will be obvious.

I ran it through valgrind (i left the command line I used in there so
you can verify the methods)

I am getting errors with valgrind inside the Alltoall function.  See
https://gist.github.com/anonymous/fbd83343f456f0688cea .

These errors do not occur in the stack allocated version.  See
https://gist.github.com/anonymous/f4dbcddbbc9fee0f508e .  I assume
this is due to valgrind being unable to detect stack corruption.

> The segfault indicates that you overwrite outside of the allocated memory 
> (and conflicts with the ptmalloc
> library). I’m quite certain that you write outside the allocated array …

So, my understanding is that Alltoall would utilize wsize multiplied
by the size of the data type indexes of the matrix and jump extent
bytes between communications.  Thus, I'm very lost as to why the
Alltoall is exceeding the bounds of my array.

Thanks,
Spenser

On Thu, May 8, 2014 at 2:19 PM, Matthieu Brucher
 wrote:
> A simple test would be to run it with valgrind, so that out of bound
> reads and writes will be obvious.
>
> Cheers,
>
> Matthieu
>
> 2014-05-08 21:16 GMT+02:00 Spenser Gilliland :
>> George & Mattheiu,
>>
>>> The Alltoall should only return when all data is sent and received on
>>> the current rank, so there shouldn't be any race condition.
>>
>> Your right this is MPI not pthreads.  That should never happen. Duh!
>>
>>> I think the issue is with the way you define the send and receive
>>> buffer in the MPI_Alltoall. You have to keep in mind that the
>>> all-to-all pattern will overwrite the entire data in the receive
>>> buffer. Thus, starting from a relative displacement in the data (in
>>> this case matrix[wrank*wrows]), begs for troubles, as you will write
>>> outside the receive buffer.
>>
>> The submatrix corresponding to matrix[wrank*wrows][0] to
>> matrix[(wrank+1)*wrows-1][:] is valid only on the wrank process.  This
>> is a block distribution of the rows like what MPI_Scatter would
>> produce.  As wrows is equal to N (matrix width/height) divided by
>> wsize, the number of mpi_all_t blocks in each message is equal to
>> wsize.  Therefore, there should be no writing outside the bounds of
>> the submatrix.
>>
>> On another note,
>> I just ported the example to use dynamic memory and now I'm getting
>> segfaults when I call MPI_Finalize().  Any idea what in the code could
>> have caused this?
>>
>> It's paste binned here: 
>> https://gist.github.com/anonymous/a80e0679c3cbffb82e39
>>
>> The result is
>>
>> [sgillila@jarvis src]$ mpirun -npernode 2 transpose2 8
>> N = 8
>> Matrix =
>>  0: 0 1 2 3 4 5 6 7
>>  0: 8 9101112131415
>>  0:1617181920212223
>>  0:2425262728293031
>>  1:3233343536373839
>>  1:4041424344454647
>>  1:4849505152535455
>>  1:5657585960616263
>> Matrix =
>>  0: 0 8162432404856
>>  0: 1 9172533414957
>>  0: 210182634425058
>>  0: 311192735435159
>>  1: 412202836445260
>>  1: 513212937455361
>>  1: 614223038465462
>>  1: 715233139475563
>> [jarvis:09314] *** Process received signal ***
>> [jarvis:09314] Signal: Segmentation fault (11)
>> [jarvis:09314] Signal code: Address not mapped (1)
>> [jarvis:09314] Failing at address: 0x21da228
>> [jarvis:09314] [ 0] /lib64/libpthread.so.0() [0x371480f500]
>> [jarvis:09314] [ 1]
>> /opt/openmpi/lib/libmpi.so.1(opal_memory_ptmalloc2_int_free+0x75)
>> [0x7f2e85452575]
>> [jarvis:09314] [ 2]
>> /opt/openmpi/lib/libmpi.so.1(opal_memory_ptmalloc2_free+0xd3)
>> [0x7f2e85452bc3]
>> [jarvis:09314] [ 3] transpose2(main+0x160) [0x4012a0]
>> [jarvis:09314] [ 4] /lib64/libc.so.6(__libc_start_main+0xfd) [0x3713c1ecdd]
>> [jarvis:09314] [ 5] transpose2() [0x400d49]
>> [jarvis:09314] *** End of error message ***
>> --
>> mpirun noticed that process rank 1 with PID 9314 on node
>> jarvis.cs.iit.edu exited on signal 11 (Segmentation fault).
>> --
>>
>> --
>> Spenser Gilliland
>> Computer Engineer
>> Doctoral Candidate
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> --
> Information System Engineer, Ph.D.
> Blog: http://matt.eifelle.com
> LinkedIn: 

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[Matthieu Brucher]

A simple test would be to run it with valgrind, so that out of bound
reads and writes will be obvious.

Cheers,

Matthieu

2014-05-08 21:16 GMT+02:00 Spenser Gilliland :
> George & Mattheiu,
>
>> The Alltoall should only return when all data is sent and received on
>> the current rank, so there shouldn't be any race condition.
>
> Your right this is MPI not pthreads.  That should never happen. Duh!
>
>> I think the issue is with the way you define the send and receive
>> buffer in the MPI_Alltoall. You have to keep in mind that the
>> all-to-all pattern will overwrite the entire data in the receive
>> buffer. Thus, starting from a relative displacement in the data (in
>> this case matrix[wrank*wrows]), begs for troubles, as you will write
>> outside the receive buffer.
>
> The submatrix corresponding to matrix[wrank*wrows][0] to
> matrix[(wrank+1)*wrows-1][:] is valid only on the wrank process.  This
> is a block distribution of the rows like what MPI_Scatter would
> produce.  As wrows is equal to N (matrix width/height) divided by
> wsize, the number of mpi_all_t blocks in each message is equal to
> wsize.  Therefore, there should be no writing outside the bounds of
> the submatrix.
>
> On another note,
> I just ported the example to use dynamic memory and now I'm getting
> segfaults when I call MPI_Finalize().  Any idea what in the code could
> have caused this?
>
> It's paste binned here: https://gist.github.com/anonymous/a80e0679c3cbffb82e39
>
> The result is
>
> [sgillila@jarvis src]$ mpirun -npernode 2 transpose2 8
> N = 8
> Matrix =
>  0: 0 1 2 3 4 5 6 7
>  0: 8 9101112131415
>  0:1617181920212223
>  0:2425262728293031
>  1:3233343536373839
>  1:4041424344454647
>  1:4849505152535455
>  1:5657585960616263
> Matrix =
>  0: 0 8162432404856
>  0: 1 9172533414957
>  0: 210182634425058
>  0: 311192735435159
>  1: 412202836445260
>  1: 513212937455361
>  1: 614223038465462
>  1: 715233139475563
> [jarvis:09314] *** Process received signal ***
> [jarvis:09314] Signal: Segmentation fault (11)
> [jarvis:09314] Signal code: Address not mapped (1)
> [jarvis:09314] Failing at address: 0x21da228
> [jarvis:09314] [ 0] /lib64/libpthread.so.0() [0x371480f500]
> [jarvis:09314] [ 1]
> /opt/openmpi/lib/libmpi.so.1(opal_memory_ptmalloc2_int_free+0x75)
> [0x7f2e85452575]
> [jarvis:09314] [ 2]
> /opt/openmpi/lib/libmpi.so.1(opal_memory_ptmalloc2_free+0xd3)
> [0x7f2e85452bc3]
> [jarvis:09314] [ 3] transpose2(main+0x160) [0x4012a0]
> [jarvis:09314] [ 4] /lib64/libc.so.6(__libc_start_main+0xfd) [0x3713c1ecdd]
> [jarvis:09314] [ 5] transpose2() [0x400d49]
> [jarvis:09314] *** End of error message ***
> --
> mpirun noticed that process rank 1 with PID 9314 on node
> jarvis.cs.iit.edu exited on signal 11 (Segmentation fault).
> --
>
> --
> Spenser Gilliland
> Computer Engineer
> Doctoral Candidate
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[Spenser Gilliland]

George & Mattheiu,

> The Alltoall should only return when all data is sent and received on
> the current rank, so there shouldn't be any race condition.

Your right this is MPI not pthreads.  That should never happen. Duh!

> I think the issue is with the way you define the send and receive
> buffer in the MPI_Alltoall. You have to keep in mind that the
> all-to-all pattern will overwrite the entire data in the receive
> buffer. Thus, starting from a relative displacement in the data (in
> this case matrix[wrank*wrows]), begs for troubles, as you will write
> outside the receive buffer.

The submatrix corresponding to matrix[wrank*wrows][0] to
matrix[(wrank+1)*wrows-1][:] is valid only on the wrank process.  This
is a block distribution of the rows like what MPI_Scatter would
produce.  As wrows is equal to N (matrix width/height) divided by
wsize, the number of mpi_all_t blocks in each message is equal to
wsize.  Therefore, there should be no writing outside the bounds of
the submatrix.

On another note,
I just ported the example to use dynamic memory and now I'm getting
segfaults when I call MPI_Finalize().  Any idea what in the code could
have caused this?

It's paste binned here: https://gist.github.com/anonymous/a80e0679c3cbffb82e39

The result is

[sgillila@jarvis src]$ mpirun -npernode 2 transpose2 8
N = 8
Matrix =
 0: 0 1 2 3 4 5 6 7
 0: 8 9101112131415
 0:1617181920212223
 0:2425262728293031
 1:3233343536373839
 1:4041424344454647
 1:4849505152535455
 1:5657585960616263
Matrix =
 0: 0 8162432404856
 0: 1 9172533414957
 0: 210182634425058
 0: 311192735435159
 1: 412202836445260
 1: 513212937455361
 1: 614223038465462
 1: 715233139475563
[jarvis:09314] *** Process received signal ***
[jarvis:09314] Signal: Segmentation fault (11)
[jarvis:09314] Signal code: Address not mapped (1)
[jarvis:09314] Failing at address: 0x21da228
[jarvis:09314] [ 0] /lib64/libpthread.so.0() [0x371480f500]
[jarvis:09314] [ 1]
/opt/openmpi/lib/libmpi.so.1(opal_memory_ptmalloc2_int_free+0x75)
[0x7f2e85452575]
[jarvis:09314] [ 2]
/opt/openmpi/lib/libmpi.so.1(opal_memory_ptmalloc2_free+0xd3)
[0x7f2e85452bc3]
[jarvis:09314] [ 3] transpose2(main+0x160) [0x4012a0]
[jarvis:09314] [ 4] /lib64/libc.so.6(__libc_start_main+0xfd) [0x3713c1ecdd]
[jarvis:09314] [ 5] transpose2() [0x400d49]
[jarvis:09314] *** End of error message ***
--
mpirun noticed that process rank 1 with PID 9314 on node
jarvis.cs.iit.edu exited on signal 11 (Segmentation fault).
--

-- 
Spenser Gilliland
Computer Engineer
Doctoral Candidate

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[George Bosilca]

I think the issue is with the way you define the send and receive
buffer in the MPI_Alltoall. You have to keep in mind that the
all-to-all pattern will overwrite the entire data in the receive
buffer. Thus, starting from a relative displacement in the data (in
this case matrix[wrank*wrows]), begs for troubles, as you will write
outside the receive buffer.

  George.


On Thu, May 8, 2014 at 10:08 AM, Matthieu Brucher
 wrote:
> The Alltoall should only return when all data is sent and received on
> the current rank, so there shouldn't be any race condition.
>
> Cheers,
>
> Matthieu
>
> 2014-05-08 15:53 GMT+02:00 Spenser Gilliland :
>> George & other list members,
>>
>> I think I may have a race condition in this example that is masked by
>> the print_matrix statement.
>>
>> For example, lets say rank one has a large sleep before reaching the
>> local transpose, will the other ranks have completed the Alltoall and
>> when rank one reaches the local transpose it is altering the data that
>> the other processors sent it?
>>
>> Thanks,
>> Spenser
>>
>>
>> --
>> Spenser Gilliland
>> Computer Engineer
>> Doctoral Candidate
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> --
> Information System Engineer, Ph.D.
> Blog: http://matt.eifelle.com
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> Music band: http://liliejay.com/
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[Matthieu Brucher]

The Alltoall should only return when all data is sent and received on
the current rank, so there shouldn't be any race condition.

Cheers,

Matthieu

2014-05-08 15:53 GMT+02:00 Spenser Gilliland :
> George & other list members,
>
> I think I may have a race condition in this example that is masked by
> the print_matrix statement.
>
> For example, lets say rank one has a large sleep before reaching the
> local transpose, will the other ranks have completed the Alltoall and
> when rank one reaches the local transpose it is altering the data that
> the other processors sent it?
>
> Thanks,
> Spenser
>
>
> --
> Spenser Gilliland
> Computer Engineer
> Doctoral Candidate
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users



-- 
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[Spenser Gilliland]

George & other list members,

I think I may have a race condition in this example that is masked by
the print_matrix statement.

For example, lets say rank one has a large sleep before reaching the
local transpose, will the other ranks have completed the Alltoall and
when rank one reaches the local transpose it is altering the data that
the other processors sent it?

Thanks,
Spenser


-- 
Spenser Gilliland
Computer Engineer
Doctoral Candidate

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[Spenser Gilliland]

George,

> Do you mind posting your working example here on the mailing list?
> This might help future users understanding how to correctly use the
> MPI datatype.

No problem. I wrote up this simplified example so others can learn to
use the functionality. This is a matrix transpose operation using MPI
vector derived types and collective communication. (Terms added to aid
search engines)

The result of running this program with a two node cluster is

Matrix =
 0: 0 1 2 3 4 5 6 7
 0: 8 9101112131415
 0:1617181920212223
 0:2425262728293031
 1:3233343536373839
 1:4041424344454647
 1:4849505152535455
 1:5657585960616263
Matrix =
 0: 0 81624 4122028
 0: 1 91725 5132129
 0: 2101826 6142230
 0: 3111927 7152331
 1:3240485636445260
 1:3341495737455361
 1:3442505838465462
 1:3543515939475563
Matrix =
 0: 0 8162432404856
 0: 1 9172533414957
 0: 210182634425058
 0: 311192735435159
 1: 412202836445260
 1: 513212937455361
 1: 614223038465462
 1: 715233139475563

/* file: transpose.c
 *
 * Description: Transpose operation using MPI derived datatypes with
 * MPI collective communications.
 *
 * Author: Spenser Gilliland 
 */

#include 
#include 
#include 

#define N (8)
float matrix[N][N];

void print_matrix(int wrank, int wrows) {
int i , j;

MPI_Barrier(MPI_COMM_WORLD);
if(wrank == 0) printf("Matrix = \n");
MPI_Barrier(MPI_COMM_WORLD);

for(i = 0; i < N; i++) {
if(i >= wrank*wrows && i < (wrank+1)*wrows) {
printf("%2d:", wrank);
for(j = 0; j < N; j++) {
printf("%6.2g", matrix[i][j]);
}
printf("\n");
}
usleep(1);
MPI_Barrier(MPI_COMM_WORLD);
}
}

int main(int argc, char *argv[]) {
int wsize, wrank, wrows;
int i, j, k;
int row, col;
float temp;
MPI_Datatype mpi_all_unaligned_t, mpi_all_t;

MPI_Init(, );
MPI_Comm_size(MPI_COMM_WORLD, );
MPI_Comm_rank(MPI_COMM_WORLD, );

wrows = N / wsize;

MPI_Type_vector(N, wrows, N, MPI_FLOAT, _all_unaligned_t);
MPI_Type_create_resized(mpi_all_unaligned_t, 0,
wrows*sizeof(float), _all_t);
MPI_Type_free(_all_unaligned_t);
MPI_Type_commit(_all_t);

for(i = 0; i < N; i++) {
/* Initialize data on the rows of the matrix owned by this rank */
if (i >= wrank*wrows && i < (wrank+1)*wrows) {
for(j = 0; j < N; j++) {
matrix[i][j] = i*N + j;
}
}
}

print_matrix(wrank, wrows);

/* Local Transpose */
row = wrank*wrows;
for(k = 0; k < wsize; k++) {
col = k*wrows;
for( i = 0; i < wrows; i++) {
for(j =i+1; j < wrows; j++) {
temp = matrix[row+i][col + j];
matrix[row + i][col + j] = matrix[row + j][col + i];
matrix[row + j][col + i] = temp;
}
}
}

print_matrix(wrank, wrows);

/* Global Transpose */
MPI_Alltoall(matrix[wrank*wrows], 1, mpi_all_t,
 matrix[wrank*wrows], 1, mpi_all_t,
 MPI_COMM_WORLD);

print_matrix(wrank, wrows);

MPI_Finalize();
return 0;
}

Thanks,
Spenser

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[George Bosilca]

Spenser,

Do you mind posting your working example here on the mailing list?
This might help future users understanding how to correctly use the
MPI datatype.

Thanks,
  George.



On Wed, May 7, 2014 at 3:16 PM, Spenser Gilliland
 wrote:
> George,
>
> Thanks for taking the time to respond to my question!  I've succeeded
> on getting my program to run using the information you provided.  I'm
> actually doing a matrix transpose with data distribute on contiguous
> rows.  However, the code I provided did not show this clearly.  Thanks
> for your insight.
>
> Thanks,
> Spenser
>
> --
> Spenser Gilliland
> Computer Engineer
> Doctoral Candidate
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[Spenser Gilliland]

George,

Thanks for taking the time to respond to my question!  I've succeeded
on getting my program to run using the information you provided.  I'm
actually doing a matrix transpose with data distribute on contiguous
rows.  However, the code I provided did not show this clearly.  Thanks
for your insight.

Thanks,
Spenser

-- 
Spenser Gilliland
Computer Engineer
Doctoral Candidate

Re: [OMPI users] MPI_Alltoall with Vector Datatype

[George Bosilca]

Spenser,

There are several issues with the code you provided.

1. You are using a 1D process grid to create a 2D block cyclic distribution. 
That’s just not possible.

2. You forgot to take in account the extent of the datatype. By default the 
extent of a vector type is starting from the first byte till the last, thus in 
your case it encompass all the data from the other processes. You have to 
resize the datatype, in order to be able to jump to the correct data for the 
next processor.

I put below a modified version of your code, that is doing something closer to 
what you expect. 

Have you look at subarray and array constructs? They might help with the 
special datatype you want, or at least they can give you a hint on how to 
correctly construct the type.

  George.

#include 
#include 

void print_matrix(float* A, int size, int wrank)
{
  int i, j;
  for(i = 0; i < size; i++) {
printf("<%2d> ", wrank);
for (j = 0; j < size; j++) {
  printf("%6.2g ", A[i*size+j]);
}
printf("\n");
  }
  printf("\n");
}

int main(int argc, char* argv[])
{
  float A[4][4];
  int i, j, wsize, wrank;
  MPI_Datatype temp, mpi_all_t;

  MPI_Init(NULL, NULL);
  MPI_Comm_size(MPI_COMM_WORLD, ); // Assume 2
  MPI_Comm_rank(MPI_COMM_WORLD, );

  MPI_Type_vector(4, 4/wsize, 4, MPI_FLOAT, );
  MPI_Type_create_resized(temp, 0, 4/wsize*sizeof(float), _all_t);
  MPI_Type_free();
  MPI_Type_commit(_all_t);

  for(i = 0; i < 4; i++)
for (j = 0; j < 4; j++) {
  A[i][j] = i*4+j;
}

  MPI_Alltoall(A, 1, mpi_all_t,
   A, 1, mpi_all_t,
   MPI_COMM_WORLD);
  print_matrix(A, 4, wrank);
  MPI_Type_free(_all_t);
  MPI_Finalize();
  return 0;
}



On May 7, 2014, at 04:48 , Spenser Gilliland  wrote:

> Hi,
> 
> I've recently started working with MPI and I noticed that when a
> Alltoall is utilized with a vector datatype, the call only uses the
> extent to determine the location for the back to back transactions.
> This makes using the vector type with collective communicators
> difficult.
> 
> For Example:
> Using the code at the bottom.  I think I should get
> 
> 0  1  8  9
> 4  5 12 13
> 2  3 10 11
> 6  7 14 15
> 
> However, the result is
> 
> 0  1  2  3
> 4  5  8  9
> 6  7 10 11
> x  x 14 15
> 
> Is this the way it is supposed to be?
> 
> FYI: This is version 1.6.2 in Rocks 6
> 
> Thanks,
> Spenser
> 
> float A[4][4];
> int wsize, wrank;
> MPI_Comm_size(MPI_COMM_WORLD, ); // Assume 2
> MPI_Comm_rank(MPI_COMM_WORLD, );
> MPI_Type_vector(4/wsize, 4/wsize, 4, MPI_FLOAT, _all_t);
> MPI_Type_commit(_all_t);
> 
> for(int i = 0; i < 4; i++) for (j = 0; j < 4; j++) {
>   A[i][j] = i*4+j;
> }
> 
> MPI_Alltoall(A[rank*4/wsize*4], 1, mpi_all_t,
> A[rank*4/wsize*4], 1, mpi_all_t,
> MPI_COMM_WORLD);
> 
> for(int i = 0; i < 4; i++) {
>   for (j = 0; j < 4; j++) {
>  printf("%6.2g")
>   }
>   printf("\n");
> }
> 
> -- 
> Spenser Gilliland
> Computer Engineer
> Doctoral Candidate
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users