Patrick,
In the worst case scenario, requiring MPI_THREAD_MULTIPLE support can
disable some fast interconnect
and make your app fallback on IPoIB or similar. And in that case, Open MPI
might prefer a suboptimal
IP network which can impact the overall performances even more.
Which threading support does your app ask?
Many applications do not call MPI in the OpenMP regions at all, or only the
master thread invokes MPI,
and in this case, MPI_THREAD_FUNNELED is enough.
Are you using UCX or the legacy openib btl?
If the former, is it built with multi threading support?
If the latter, I suggest you give UCX - built with multi threading support
- a try and see how it goes
Cheers,
Gilles
On Thu, Mar 24, 2022 at 5:43 PM Patrick Begou via users <
users@lists.open-mpi.org> wrote:
> Le 28/02/2022 à 17:56, Patrick Begou via users a écrit :
> > Hi,
> >
> > I meet a performance problem with OpenMPI on my cluster. In some
> > situation my parallel code is really slow (same binary running on a
> > different mesh).
> >
> > To investigate, the fortran code code is built with profiling option
> > (mpifort -p -O3.) and launched on 91 cores.
> >
> > One mon.out file per process, they show a maximum cpu time of 20.4
> > seconds for each processes (32.7 seconds on my old cluster) and this
> > is Ok.
> >
> > But running on my new cluster requires near 3mn instead of 1mn on the
> > old cluster (elapsed time).
> >
> > New cluster is running OpenMPI 4.05 with HDR-100 connections.
> >
> > Old cluster is running OpenMPI 3.1 with QDR connections.
> >
> > Running Osu Collectives tests on 91 cores shows good latency values on
> > 91 cores and the point-to-points between nodes is correct.
> >
> > How can I investigate this problem as it seams related to MPI
> > communications in some situations that I can reproduce? Using Scalasca
> > ? Other tools ? OpenMPI is not built with special profiling options.
> >
> > Thanks
> >
> > Patrick
> >
> >
> Just to provide an answer to this old thread, the problem has been found
> (but not solved). The application was rebuilt with OpenMP flag (hybrid
> parallelism is implemented with MPI and OpenMP). Setting this flag, even
> if we only use one thread and MPI only parallelism, change OpenMPI
> initialisation from MPI_INIT to MPI_INIT_THREAD in our code and this
> create the big slowdown of the application.
>
> We have temporally removed the OpenMP flag to build the application.
>
> Patrick
>
>
>
