Sam,
Open MPI aggregates messages only when network constraints prevent the
messages from being timely delivered. In this particular case I think that
our delayed business card exchange and connection setup is delaying the
delivery of the first batch of messages (and the BTL will aggregate them
while waiting for the connection to be correctly setup).
Can you reproduce the same behavior after the first batch of messages ?
Assuming the times showed on the left of your messages are correct, the
first MPI seems to deliver the entire set of messages significantly faster
than the second.
George.
On Thu, Jul 20, 2017 at 5:42 PM, Samuel Thibault <samuel.thiba...@labri.fr>
wrote:
> Hello,
>
> We are getting a strong performance issue, which is due to a missing
> pipelining behavior from OpenMPI when running over TCP. I have attached
> a test case. Basically what it does is
>
> if (myrank == 0) {
> for (i = 0; i < N; i++)
> MPI_Isend(...);
> } else {
> for (i = 0; i < N; i++)
> MPI_Irecv(...);
> }
> for (i = 0; i < N; i++)
> MPI_Wait(...);
>
> with corresponding printfs. And the result is:
>
> 0.182620: Isend 0 begin
> 0.182761: Isend 0 end
> 0.182766: Isend 1 begin
> 0.182782: Isend 1 end
> ...
> 0.183911: Isend 49 begin
> 0.183915: Isend 49 end
> 0.199028: Irecv 0 begin
> 0.199068: Irecv 0 end
> 0.199070: Irecv 1 begin
> 0.199072: Irecv 1 end
> ...
> 0.199187: Irecv 49 begin
> 0.199188: Irecv 49 end
> 0.233948: Isend 0 done!
> 0.269895: Isend 1 done!
> ...
> 1.982475: Isend 49 done!
> 1.984065: Irecv 0 done!
> 1.984078: Irecv 1 done!
> ...
> 1.984131: Irecv 49 done!
>
> i.e. almost two seconds happen between the start of the application and
> the first Irecv completes, and then all Irecv complete immediately too,
> i.e. it seems the communications were grouped altogether.
>
> This is really bad, because in our real use case, we trigger
> computations after each MPI_Wait calls, and we use several messages so
> as to pipeline things: the first computation can start as soon as one
> message gets received, thus overlapped with further receptions.
>
> This problem is only with openmpi on TCP, I'm not getting this behavior
> with openmpi on IB, and I'm not getting it either with mpich or madmpi:
>
> 0.182168: Isend 0 begin
> 0.182235: Isend 0 end
> 0.182237: Isend 1 begin
> 0.182242: Isend 1 end
> ...
> 0.182842: Isend 49 begin
> 0.182844: Isend 49 end
> 0.200505: Irecv 0 begin
> 0.200564: Irecv 0 end
> 0.200567: Irecv 1 begin
> 0.200569: Irecv 1 end
> ...
> 0.201233: Irecv 49 begin
> 0.201234: Irecv 49 end
> 0.269511: Isend 0 done!
> 0.273154: Irecv 0 done!
> 0.341054: Isend 1 done!
> 0.344507: Irecv 1 done!
> ...
> 3.767726: Isend 49 done!
> 3.770637: Irecv 49 done!
>
> There we do have pipelined reception.
>
> Is there a way to get the second, pipelined behavior with openmpi on
> TCP?
>
> Samuel
>
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
