Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04?
Are you using gcc provided by Ubuntu 20.04?
if not which compiler (vendor and version) are you using?
My (light) understanding is that this patch should not impact
performances, so I am not
sure whether the performance being back is something I do not understand,
or the side effect
of a compiler bug.
Anyway, I issued https://github.com/open-mpi/ompi/pull/8789 and asked
for a review.
Cheers,
Gilles
- Original Message -
> Dear Gilles,
> As per your suggestion, I tried the inline patch
as discussed in
https://github.com/open-mpi/ompi/pull/8622#issuecomment-800776864
.
>
> This has fixed the regression completely for the remaining test cases
in FFTW MPI in-built test bench - which was persisting even after using
the git patch
https://patch-diff.githubusercontent.com/raw/open-mpi/ompi/pull/8623.patch
as merged by you.
> So, it seems there is a performance difference between asm volatile("":
: :"memory"); and __atomic_thread_fence (__ATOMIC_ACQUIRE) on x86_64.
>
> I would request you to please make this change and merge it to
respective openMPI branches - please intimate if possible whenever that
takes place.
> I also request you to plan for an early 4.1.1rc2 release at least by
June 2021.
>
> With Regards,
> S. Biplab Raut
>
> -Original Message-
> From: Gilles Gouaillardet
> Sent: Thursday, April 1, 2021 8:31 AM
> To: Raut, S Biplab
> Subject: Re: [OMPI users] Stable and performant openMPI version for
Ubuntu20.04 ?
>
> [CAUTION: External Email]
>
> I really had no time to investigate this.
>
> A quick test is to apply the patch in the inline comment at
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopen-mpi%2Fompi%2Fpull%2F8622%23issuecomment-800776864data=04%7C01%7CBiplab.Raut%40amd.com%7C6b277b24afa04650c86c08d8f4ba5dc7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637528428572315404%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=4PBWIZsFdyBO2gUbYURh9iDwQxMdM%2FUfQV4%2Bg%2Farnh0%3Dreserved=0
and see whether it helps.
>
> If not, I would recommend you try Open MPI 3.1.6 (after manually
applying
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopen-mpi%2Fompi%2Fpull%2F8624.patchdata=04%7C01%7CBiplab.Raut%40amd.com%7C6b277b24afa04650c86c08d8f4ba5dc7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637528428572315404%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=yZbu1dDcC1awpiuclvyso9HANqAHIEn4p1pT862n4LY%3Dreserved=0)
and see whether there is a performance regression between 3.1.1 and (
patched) 3.1.6
>
> Cheers,
>
> Gilles
>
> On Thu, Apr 1, 2021 at 11:25 AM Raut, S Biplab
wrote:
> >
> > Dear Gilles,
> > Did you get a chance to look into my below mail
content?
> > I find the regression is not completely fixed.
> >
> > With Regards,
> > S. Biplab Raut
> >
> > -Original Message-
> > From: Raut, S Biplab
> > Sent: Wednesday, March 24, 2021 11:32 PM
> > To: Gilles Gouaillardet
> > Subject: RE: [OMPI users] Stable and performant openMPI version for
Ubuntu20.04 ?
> >
> > Dear Gilles,
> > After applying the below patch, I thoroughly
tested various test cases of FFTW using its in-built benchmark test
program.
> > Many of the test cases, that showed regression previously as
compared to openMPI3.1.1, have now improved with positive gains.
> > However, there are still few test cases where the performance is
lower than openMPI3.1.1.
> > Are there more performance issues in openMPI4.x that need to be
discovered?
> >
> > Please check the below details.
> >
> > 1) For problem size 1024x1024x512 :-
> > $ mpirun --map-by core --rank-by core --bind-to core ./fftw/
mpi/mpi-bench -opatient -r500 -s dcif1024x1024x512
> > openMPI3.3.1_stock performance -> 147 MFLOPS
> > openMPI4.1.0_stock performance -> 137 MFLOPS
> > openMPI4.1.0_patch performance -> 137 MFLOPS
> > 2) For problem size 512x512x512 :-
> > $ mpirun --map-by core --rank-by core --bind-to core ./fftw/
mpi/mpi-bench -opatient -r500 -s dcif512x512x512
> > openMPI3.3.1_stock performance -> 153 MFLOPS
> > openMPI4.1.0_stock performance -> 144 MFLOPS
> > openMPI4.1.0_patch performance -> 147 MFLOPS
> >
> > With Regards,
> > S. Biplab Rsut
> >
> > -Original Message-
> > From: Gilles Gouaillardet
> > Sent: Wednesday, March 17, 2021 11:14 AM
> > To: Raut, S Biplab
> > Subject: Re: [OMPI users] Stable and performant openMPI version for
Ubuntu20.04 ?
> >
> > [CAUTION: External Email]
> >
> > The patch has been merged into the v4.1.x release branch, but 4.1.
1rc2 has not been yet released.
> > Your best bet is to download and apply the patch at
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> > ub.com%2Fopen-mpi%2Fompi%2Fpull%2F8623.patchdata=04%7C01%
7CBiplab
> >
Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?
Dear Gilles,
As per your suggestion, I tried the inline patch as
discussed in https://github.com/open-mpi/ompi/pull/8622#issuecomment-800776864 .
This has fixed the regression completely for the remaining test cases in FFTW
MPI in-built test bench - which was persisting even after using the git patch
https://patch-diff.githubusercontent.com/raw/open-mpi/ompi/pull/8623.patch as
merged by you.
So, it seems there is a performance difference between asm volatile("": :
:"memory"); and __atomic_thread_fence (__ATOMIC_ACQUIRE) on x86_64.
I would request you to please make this change and merge it to respective
openMPI branches - please intimate if possible whenever that takes place.
I also request you to plan for an early 4.1.1rc2 release at least by June 2021.
With Regards,
S. Biplab Raut
-Original Message-
From: Gilles Gouaillardet
Sent: Thursday, April 1, 2021 8:31 AM
To: Raut, S Biplab
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04
?
[CAUTION: External Email]
I really had no time to investigate this.
A quick test is to apply the patch in the inline comment at
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopen-mpi%2Fompi%2Fpull%2F8622%23issuecomment-800776864data=04%7C01%7CBiplab.Raut%40amd.com%7C6b277b24afa04650c86c08d8f4ba5dc7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637528428572315404%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=4PBWIZsFdyBO2gUbYURh9iDwQxMdM%2FUfQV4%2Bg%2Farnh0%3Dreserved=0
and see whether it helps.
If not, I would recommend you try Open MPI 3.1.6 (after manually applying
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopen-mpi%2Fompi%2Fpull%2F8624.patchdata=04%7C01%7CBiplab.Raut%40amd.com%7C6b277b24afa04650c86c08d8f4ba5dc7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637528428572315404%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=yZbu1dDcC1awpiuclvyso9HANqAHIEn4p1pT862n4LY%3Dreserved=0)
and see whether there is a performance regression between 3.1.1 and (patched)
3.1.6
Cheers,
Gilles
On Thu, Apr 1, 2021 at 11:25 AM Raut, S Biplab wrote:
>
> Dear Gilles,
> Did you get a chance to look into my below mail content?
> I find the regression is not completely fixed.
>
> With Regards,
> S. Biplab Raut
>
> -Original Message-
> From: Raut, S Biplab
> Sent: Wednesday, March 24, 2021 11:32 PM
> To: Gilles Gouaillardet
> Subject: RE: [OMPI users] Stable and performant openMPI version for
> Ubuntu20.04 ?
>
> Dear Gilles,
> After applying the below patch, I thoroughly tested
> various test cases of FFTW using its in-built benchmark test program.
> Many of the test cases, that showed regression previously as compared to
> openMPI3.1.1, have now improved with positive gains.
> However, there are still few test cases where the performance is lower than
> openMPI3.1.1.
> Are there more performance issues in openMPI4.x that need to be discovered?
>
> Please check the below details.
>
> 1) For problem size 1024x1024x512 :-
> $ mpirun --map-by core --rank-by core --bind-to core
> ./fftw/mpi/mpi-bench -opatient -r500 -s dcif1024x1024x512
> openMPI3.3.1_stock performance -> 147 MFLOPS
> openMPI4.1.0_stock performance -> 137 MFLOPS
> openMPI4.1.0_patch performance -> 137 MFLOPS
> 2) For problem size 512x512x512 :-
> $ mpirun --map-by core --rank-by core --bind-to core
> ./fftw/mpi/mpi-bench -opatient -r500 -s dcif512x512x512
> openMPI3.3.1_stock performance -> 153 MFLOPS
> openMPI4.1.0_stock performance -> 144 MFLOPS
> openMPI4.1.0_patch performance -> 147 MFLOPS
>
> With Regards,
> S. Biplab Rsut
>
> -Original Message-
> From: Gilles Gouaillardet
> Sent: Wednesday, March 17, 2021 11:14 AM
> To: Raut, S Biplab
> Subject: Re: [OMPI users] Stable and performant openMPI version for
> Ubuntu20.04 ?
>
> [CAUTION: External Email]
>
> The patch has been merged into the v4.1.x release branch, but 4.1.1rc2 has
> not been yet released.
> Your best bet is to download and apply the patch at
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fopen-mpi%2Fompi%2Fpull%2F8623.patchdata=04%7C01%7CBiplab
> .Raut%40amd.com%7C6b277b24afa04650c86c08d8f4ba5dc7%7C3dd8961fe4884e608
> e11a82d994e183d%7C0%7C0%7C637528428572315404%7CUnknown%7CTWFpbGZsb3d8e
> yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C10
> 00sdata=OKGqhhQM68vPhuADfVdqOlHmY0ZHGtUdM%2B1WeeJ9WoY%3Drese
> rved=0 (since this does not involve any configury stuff, the process
> should be painless)
>
> Cheers,
>
> Gilles
>
> On Wed, Mar 17, 2021 at 2:31 PM Raut, S Biplab wrote:
> >
> > Dear Gilles,
> > Thank you for your support and quick fix for this
> > issue.
> > Could you tell me if the fix is finally merged and how do
