Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-04-09 Thread Raut, S Biplab via users 
Dear Gilles,
I am using GCC (v9.3.0) as provided by Ubuntu 20.04.
As I said only few test cases were in regression with the earlier patch, but 
this new patch resolves all of them.

With Regards,
S. Biplab Raut

-Original Message-
From: gil...@rist.or.jp  
Sent: Friday, April 9, 2021 5:45 AM
To: Open MPI Users 
Cc: Gilles Gouaillardet ; Raut, S Biplab 

Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?

[CAUTION: External Email]

Are you using gcc provided by Ubuntu 20.04?
if not which compiler (vendor and version) are you using?

My (light) understanding is that this patch should not impact performances, so 
I am not sure whether the performance being back is something I do not 
understand,  or the side effect of a compiler bug.

Anyway, I issued 
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopen-mpi%2Fompi%2Fpull%2F8789data=04%7C01%7CBiplab.Raut%40amd.com%7C1c4ece784e9046362baa08d8faec8ebf%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637535241229925553%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=BDCXkCjWmem3ajzvqY%2FbHygOytd5GzUxRz8ZuEofLLk%3Dreserved=0
 and asked for a review.

Cheers,

Gilles

- Original Message -
> Dear Gilles,
> As per your suggestion, I tried the inline patch
as discussed in 
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopen-mpi%2Fompi%2Fpull%2F8622%23issuecomment-800776864data=04%7C01%7CBiplab.Raut%40amd.com%7C1c4ece784e9046362baa08d8faec8ebf%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637535241229925553%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=M2zHaN2MQ1y0XpXjGzorrJ7YvTTV0RhnLjtP66GTd3s%3Dreserved=0
 .
>
> This has fixed the regression completely for the remaining test cases
in FFTW MPI in-built test bench - which was persisting even after using the git 
patch 
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatch-diff.githubusercontent.com%2Fraw%2Fopen-mpi%2Fompi%2Fpull%2F8623.patchdata=04%7C01%7CBiplab.Raut%40amd.com%7C1c4ece784e9046362baa08d8faec8ebf%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637535241229925553%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=jheNkQa3d177uME1Qtxc9y4GTsOKyaAOgyC9%2Bqu0hcU%3Dreserved=0
 as merged by you.
> So, it seems there is a performance difference between asm volatile("":
 : :"memory"); and __atomic_thread_fence (__ATOMIC_ACQUIRE) on x86_64.
>
> I would request you to please make this change and merge it to
respective openMPI branches - please intimate if possible whenever that takes 
place.
> I also request you to plan for an early 4.1.1rc2 release at least by
June 2021.
>
> With Regards,
> S. Biplab Raut
>
> -Original Message-
> From: Gilles Gouaillardet 
> Sent: Thursday, April 1, 2021 8:31 AM
> To: Raut, S Biplab 
> Subject: Re: [OMPI users] Stable and performant openMPI version for
Ubuntu20.04 ?
>
> [CAUTION: External Email]
>
> I really had no time to investigate this.
>
> A quick test is to apply the patch in the inline comment at
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fopen-mpi%2Fompi%2Fpull%2F8622%23issuecomment-800776864da
> ta=04%7C01%7CBiplab.Raut%40amd.com%7C1c4ece784e9046362baa08d8faec8ebf%
> 7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637535241229925553%7CUnkn
> own%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwi
> LCJXVCI6Mn0%3D%7C1000sdata=M2zHaN2MQ1y0XpXjGzorrJ7YvTTV0RhnLjtP66
> GTd3s%3Dreserved=0
 and see whether it helps.
>
> If not, I would recommend you try Open MPI 3.1.6 (after manually
applying 
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopen-mpi%2Fompi%2Fpull%2F8624.patchdata=04%7C01%7CBiplab.Raut%40amd.com%7C1c4ece784e9046362baa08d8faec8ebf%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637535241229925553%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=FxcMmtniXES2%2BWaRijQf2T5GQFupDqOv5ytZvjg2NLo%3Dreserved=0)
 and see whether there is a performance regression between 3.1.1 and (
patched) 3.1.6
>
> Cheers,
>
> Gilles
>
> On Thu, Apr 1, 2021 at 11:25 AM Raut, S Biplab 
wrote:
> >
> > Dear Gilles,
> >  Did you get a chance to look into my below mail
content?
> > I find the regression is not completely fixed.
> >
> > With Regards,
> > S. Biplab Raut
> >
> > -Original Message-
> > From: Raut, S Biplab
> > Sent: Wednesday, March 24, 2021 11:32 PM
> > To: Gilles Gouaillardet 
> > Subject: RE: [OMPI users] Stable and performant openMPI version for
Ubuntu20.04 ?
> >
> &

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04?

2021-04-08 Thread Gilles Gouaillardet via users 
Are you using gcc provided by Ubuntu 20.04?
if not which compiler (vendor and version) are you using?

My (light) understanding is that this patch should not impact 
performances, so I am not
sure whether the performance being back is something I do not understand,
 or the side effect
of a compiler bug.

Anyway, I issued https://github.com/open-mpi/ompi/pull/8789 and asked 
for a review.

Cheers,

Gilles

- Original Message -
> Dear Gilles,
> As per your suggestion, I tried the inline patch 
as discussed in 
https://github.com/open-mpi/ompi/pull/8622#issuecomment-800776864
 .
> 
> This has fixed the regression completely for the remaining test cases 
in FFTW MPI in-built test bench - which was persisting even after using 
the git patch 
https://patch-diff.githubusercontent.com/raw/open-mpi/ompi/pull/8623.patch
 as merged by you.
> So, it seems there is a performance difference between asm volatile("":
 : :"memory"); and __atomic_thread_fence (__ATOMIC_ACQUIRE) on x86_64.
> 
> I would request you to please make this change and merge it to 
respective openMPI branches - please intimate if possible whenever that 
takes place.
> I also request you to plan for an early 4.1.1rc2 release at least by 
June 2021.
> 
> With Regards,
> S. Biplab Raut 
> 
> -Original Message-
> From: Gilles Gouaillardet  
> Sent: Thursday, April 1, 2021 8:31 AM
> To: Raut, S Biplab 
> Subject: Re: [OMPI users] Stable and performant openMPI version for 
Ubuntu20.04 ?
> 
> [CAUTION: External Email]
> 
> I really had no time to investigate this.
> 
> A quick test is to apply the patch in the inline comment at
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopen-mpi%2Fompi%2Fpull%2F8622%23issuecomment-800776864data=04%7C01%7CBiplab.Raut%40amd.com%7C6b277b24afa04650c86c08d8f4ba5dc7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637528428572315404%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=4PBWIZsFdyBO2gUbYURh9iDwQxMdM%2FUfQV4%2Bg%2Farnh0%3Dreserved=0
 and see whether it helps.
> 
> If not, I would recommend you try Open MPI 3.1.6 (after manually 
applying 
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopen-mpi%2Fompi%2Fpull%2F8624.patchdata=04%7C01%7CBiplab.Raut%40amd.com%7C6b277b24afa04650c86c08d8f4ba5dc7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637528428572315404%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=yZbu1dDcC1awpiuclvyso9HANqAHIEn4p1pT862n4LY%3Dreserved=0)
 and see whether there is a performance regression between 3.1.1 and (
patched) 3.1.6
> 
> Cheers,
> 
> Gilles
> 
> On Thu, Apr 1, 2021 at 11:25 AM Raut, S Biplab  
wrote:
> >
> > Dear Gilles,
> >  Did you get a chance to look into my below mail 
content?
> > I find the regression is not completely fixed.
> >
> > With Regards,
> > S. Biplab Raut
> >
> > -Original Message-----
> > From: Raut, S Biplab
> > Sent: Wednesday, March 24, 2021 11:32 PM
> > To: Gilles Gouaillardet 
> > Subject: RE: [OMPI users] Stable and performant openMPI version for 
Ubuntu20.04 ?
> >
> > Dear Gilles,
> > After applying the below patch, I thoroughly 
tested various test cases of FFTW using its in-built benchmark test 
program.
> > Many of the test cases, that showed regression previously as 
compared to openMPI3.1.1, have now improved with positive gains.
> > However, there are still few test cases where the performance is 
lower than openMPI3.1.1.
> > Are there more performance issues in openMPI4.x that need to be 
discovered?
> >
> > Please check the below details.
> >
> > 1) For problem size 1024x1024x512 :-
> >  $   mpirun --map-by core --rank-by core --bind-to core  ./fftw/
mpi/mpi-bench -opatient -r500 -s dcif1024x1024x512
> >  openMPI3.3.1_stock performance -> 147 MFLOPS
> >  openMPI4.1.0_stock performance -> 137 MFLOPS
> >  openMPI4.1.0_patch performance -> 137 MFLOPS
> > 2) For problem size 512x512x512 :-
> >  $   mpirun --map-by core --rank-by core --bind-to core  ./fftw/
mpi/mpi-bench -opatient -r500 -s dcif512x512x512
> >  openMPI3.3.1_stock performance -> 153  MFLOPS
> >      openMPI4.1.0_stock performance -> 144 MFLOPS
> >  openMPI4.1.0_patch performance -> 147 MFLOPS
> >
> > With Regards,
> > S. Biplab Rsut
> >
> > -Original Message-
> > From: Gilles Gouaillardet 
> > Sent: Wednesday, March 17, 2021 11:14 AM
> > To: Raut, S Biplab 
> > Subject: Re: [OMPI users] Stable and performant openMPI version fo

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-04-08 Thread Raut, S Biplab via users 
Dear Gilles,
As per your suggestion, I tried the inline patch as 
discussed in https://github.com/open-mpi/ompi/pull/8622#issuecomment-800776864 .

This has fixed the regression completely for the remaining test cases in FFTW 
MPI in-built test bench - which was persisting even after using the git patch 
https://patch-diff.githubusercontent.com/raw/open-mpi/ompi/pull/8623.patch as 
merged by you.
So, it seems there is a performance difference between asm volatile("": : 
:"memory"); and __atomic_thread_fence (__ATOMIC_ACQUIRE) on x86_64.

I would request you to please make this change and merge it to respective 
openMPI branches - please intimate if possible whenever that takes place.
I also request you to plan for an early 4.1.1rc2 release at least by June 2021.

With Regards,
S. Biplab Raut 

-Original Message-
From: Gilles Gouaillardet  
Sent: Thursday, April 1, 2021 8:31 AM
To: Raut, S Biplab 
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?

[CAUTION: External Email]

I really had no time to investigate this.

A quick test is to apply the patch in the inline comment at
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopen-mpi%2Fompi%2Fpull%2F8622%23issuecomment-800776864data=04%7C01%7CBiplab.Raut%40amd.com%7C6b277b24afa04650c86c08d8f4ba5dc7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637528428572315404%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=4PBWIZsFdyBO2gUbYURh9iDwQxMdM%2FUfQV4%2Bg%2Farnh0%3Dreserved=0
 and see whether it helps.

If not, I would recommend you try Open MPI 3.1.6 (after manually applying 
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopen-mpi%2Fompi%2Fpull%2F8624.patchdata=04%7C01%7CBiplab.Raut%40amd.com%7C6b277b24afa04650c86c08d8f4ba5dc7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637528428572315404%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=yZbu1dDcC1awpiuclvyso9HANqAHIEn4p1pT862n4LY%3Dreserved=0)
 and see whether there is a performance regression between 3.1.1 and (patched) 
3.1.6

Cheers,

Gilles

On Thu, Apr 1, 2021 at 11:25 AM Raut, S Biplab  wrote:
>
> Dear Gilles,
>  Did you get a chance to look into my below mail content?
> I find the regression is not completely fixed.
>
> With Regards,
> S. Biplab Raut
>
> -Original Message-
> From: Raut, S Biplab
> Sent: Wednesday, March 24, 2021 11:32 PM
> To: Gilles Gouaillardet 
> Subject: RE: [OMPI users] Stable and performant openMPI version for 
> Ubuntu20.04 ?
>
> Dear Gilles,
> After applying the below patch, I thoroughly tested 
> various test cases of FFTW using its in-built benchmark test program.
> Many of the test cases, that showed regression previously as compared to 
> openMPI3.1.1, have now improved with positive gains.
> However, there are still few test cases where the performance is lower than 
> openMPI3.1.1.
> Are there more performance issues in openMPI4.x that need to be discovered?
>
> Please check the below details.
>
> 1) For problem size 1024x1024x512 :-
>  $   mpirun --map-by core --rank-by core --bind-to core  
> ./fftw/mpi/mpi-bench -opatient -r500 -s dcif1024x1024x512
>  openMPI3.3.1_stock performance -> 147 MFLOPS
>  openMPI4.1.0_stock performance -> 137 MFLOPS
>  openMPI4.1.0_patch performance -> 137 MFLOPS
> 2) For problem size 512x512x512 :-
>  $   mpirun --map-by core --rank-by core --bind-to core  
> ./fftw/mpi/mpi-bench -opatient -r500 -s dcif512x512x512
>  openMPI3.3.1_stock performance -> 153  MFLOPS
>  openMPI4.1.0_stock performance -> 144 MFLOPS
>  openMPI4.1.0_patch performance -> 147 MFLOPS
>
> With Regards,
> S. Biplab Rsut
>
> -----Original Message-
> From: Gilles Gouaillardet 
> Sent: Wednesday, March 17, 2021 11:14 AM
> To: Raut, S Biplab 
> Subject: Re: [OMPI users] Stable and performant openMPI version for 
> Ubuntu20.04 ?
>
> [CAUTION: External Email]
>
> The patch has been merged into the v4.1.x release branch, but 4.1.1rc2 has 
> not been yet released.
> Your best bet is to download and apply the patch at
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fopen-mpi%2Fompi%2Fpull%2F8623.patchdata=04%7C01%7CBiplab
> .Raut%40amd.com%7C6b277b24afa04650c86c08d8f4ba5dc7%7C3dd8961fe4884e608
> e11a82d994e183d%7C0%7C0%7C637528428572315404%7CUnknown%7CTWFpbGZsb3d8e
> yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C10
> 00sdata=OKGqhhQM68vPhuADfVdqOlHmY0ZHGtUdM%2B1WeeJ9WoY%3Drese
> rved=0 (since this does not involve any configury stuff, the process 
> should be painless)
>
> Cheers,
>
>

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-14 Thread Raut, S Biplab via users 
Dear Gilles,

 Reposting your comments along with my replies in the 
mailing-list for everybody to view/react.



I am seeing some important performance degradation between Open MPI

3.1.1 and the top of the v3.1.x branch

when running on a large number of cores.

Same performance between 4.1.0 and the top of v3.1.x

I am now running git bisect to find out when this started happening.

I am finally feeling relieved and happy that you could reproduce and 
acknowledge this regression !!

Do I need to file any bug officially anywhere?



IIRC, I noted an xpmem error in your logs (that means xpmem is not used).

The root cause could be that the xpmem kernel module is not loaded, of the 
permissions on the device are incorrect As Nathan pointed out, xpmem is likely 
to get the best performances, so while I am running git bisect, I do invite you 
to fix your xpmem issue and see how this impacts performances

Sure, I will try to fix the xpmem error and check the impact on the performance.



With Regards,

S. Biplab Raut



-Original Message-
From: Gilles Gouaillardet 
mailto:gilles.gouaillar...@gmail.com>>
Sent: Sunday, March 14, 2021 8:45 AM
To: Raut, S Biplab mailto:biplab.r...@amd.com>>
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?



[CAUTION: External Email]



I am seeing some important performance degradation between Open MPI

3.1.1 and the top of the v3.1.x branch

when running on a large number of cores.

Same performance between 4.1.0 and the top of v3.1.x



I am now running git bisect to find out when this started happening.



IIRC, I noted an xpmem error in your logs (that means xpmem is not used).

The root cause could be that the xpmem kernel module is not loaded, of the 
permissions on the device are incorrect As Nathan pointed out, xpmem is likely 
to get the best performances, so while I am running git bisect, I do invite you 
to fix your xpmem issue and see how this impacts performances



Cheers,



Gilles



On Sat, Mar 13, 2021 at 12:08 AM Raut, S Biplab 
mailto:biplab.r...@amd.com>> wrote:

>

> Dear Gilles,

>

> Please check my replies inline.

>

>

>

> >>> Can you please post the output of

>

> >>> ompi_info --param btl vader --level 3

>

> >>> with both Open MPI 3.1 and 4.1?

>

> openMPI3.1.1

>

> --

>

> $ ompi_info --param btl vader --level 3

>

>  MCA btl: vader (MCA v2.1.0, API v3.0.0, Component

> v3.1.1)

>

>MCA btl vader:

> ---

>

>MCA btl vader: parameter "btl_vader_single_copy_mechanism"

>

>   (current value: "cma", data source: default, level:

>

>   3 user/all, type: int)

>

>   Single copy mechanism to use (defaults to

> best

>

>   available)

>

>   Valid values: 1:"cma", 3:"none"

>

> openMPI4.1.0

>

> --

>

> $ ompi_info --param btl vader --level 3

>

>  MCA btl: vader (MCA v2.1.0, API v3.1.0, Component

> v4.1.0)

>

>MCA btl vader:

> ---

>

>MCA btl vader: parameter "btl_vader_single_copy_mechanism"

>

>   (current value: "cma", data source: default, level:

>

>   3 user/all, type: int)

>

>   Single copy mechanism to use (defaults to

> best

>

>   available)

>

>   Valid values: 1:"cma", 4:"emulated", 3:"none"

>

>MCA btl vader: parameter "btl_vader_backing_directory"

> (current

>

>   value: "/dev/shm", data source: default,

> level: 3

>

>   user/all, type: string)

>

>   Directory to place backing files for shared

> memory

>

>   communication. This directory should be on a

> local

>

>   filesystem such as /tmp or /dev/shm (default:

>

>   (linux) /dev/shm, (others) session

> directory)

>

>

>

> >>> What if you run with only 2 MPI ranks?

>

> >>> do you observe similar performance differences between Open MPI 3.1 and 
> >>> 4.1?

>

> When I run only 2 MPI ranks, the performance regression is not significant.

>

> openMPI3.1.1 gives MFLOPS: 11122

>

> openMPI4.1.0 gives MFLOPS: 11041

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-12 Thread Raut, S Biplab via users 
Dear Gilles,
Please check my replies inline.

>>> Can you please post the output of
>>> ompi_info --param btl vader --level 3
>>> with both Open MPI 3.1 and 4.1?
openMPI3.1.1
--
$ ompi_info --param btl vader --level 3
 MCA btl: vader (MCA v2.1.0, API v3.0.0, Component v3.1.1)
   MCA btl vader: ---
   MCA btl vader: parameter "btl_vader_single_copy_mechanism"
  (current value: "cma", data source: default, level:
  3 user/all, type: int)
  Single copy mechanism to use (defaults to best
  available)
  Valid values: 1:"cma", 3:"none"
openMPI4.1.0
--
$ ompi_info --param btl vader --level 3
 MCA btl: vader (MCA v2.1.0, API v3.1.0, Component v4.1.0)
   MCA btl vader: ---
   MCA btl vader: parameter "btl_vader_single_copy_mechanism"
  (current value: "cma", data source: default, level:
  3 user/all, type: int)
  Single copy mechanism to use (defaults to best
  available)
  Valid values: 1:"cma", 4:"emulated", 3:"none"
   MCA btl vader: parameter "btl_vader_backing_directory" (current
  value: "/dev/shm", data source: default, level: 3
  user/all, type: string)
  Directory to place backing files for shared memory
  communication. This directory should be on a local
  filesystem such as /tmp or /dev/shm (default:
  (linux) /dev/shm, (others) session directory)

>>> What if you run with only 2 MPI ranks?
>>> do you observe similar performance differences between Open MPI 3.1 and 4.1?
When I run only 2 MPI ranks, the performance regression is not significant.
openMPI3.1.1 gives MFLOPS: 11122
openMPI4.1.0 gives MFLOPS: 11041

With Regards,
S. Biplab Raut

From: Gilles Gouaillardet 
Sent: Friday, March 12, 2021 7:07 PM
To: Raut, S Biplab 
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?

[CAUTION: External Email]
Can you please post the output of

ompi_info --param btl vader --level 3

with both Open MPI 3.1 and 4.1?



What if you run with only 2 MPI ranks?

do you observe similar performance differences between Open MPI 3.1 and 4.1?



Cheers,

Gilles

On Fri, Mar 12, 2021 at 6:31 PM Raut, S Biplab 
mailto:biplab.r...@amd.com>> wrote:
Dear Gilles,
Thank you for the reply.

>>> when running
>>> mpirun --map-by core -rank-by core --bind-to core --mca pml ob1 --mca btl 
>>> vader,self ./mpi-bench ic100
>>> I go similar flops with Open MPI 3.1.1, 3.1.6, 4.1.0 and 4.1.1rc1 on my 
>>> system
>>> If you are using a different command line, please let me know and I will 
>>> give it a try
Although the command line that I use is different, but I ran with the above 
command line as used by you.
I still find that openMPI4.1.0 is poor as compared to openMPI3.1.1. Please 
check the details below. I have also provided my system details if it matters.
openMPI3.1.1
---
$ mpirun --map-by core -rank-by core --bind-to core --mca pml ob1 --mca btl 
vader,self ./mpi-bench ic100
Problem: ic100, setup: 552.20 ms, time: 1.33 ms, ``mflops'': 75143
$ ompi_info --all|grep 'command line'
  Configure command line: '--prefix=/home/server/ompi3/gcc' 
'--enable-mpi-fortran' '--enable-mpi-cxx' '--enable-shared=yes' 
'--enable-static=yes' '--enable-mpi1-compatibility'
  User-specified command line parameters passed to 
ROMIO's configure script
  Complete set of command line parameters passed to 
ROMIO's configure script

openMPI4.1.0
---
$ mpirun --map-by core -rank-by core --bind-to core --mca pml ob1 --mca btl 
vader,self ./mpi-bench ic100
Problem: ic100, setup: 557.12 ms, time: 1.75 ms, ``mflops'': 57029
$ ompi_info --all|grep 'command line'
  Configure command line: '--prefix=/home/server/ompi4_plain' 
'--enable-mpi-fortran' '--enable-mpi-cxx' '--enable-shared=yes' 
'--enable-static=yes' '--enable-mpi1-compatibility'
  User-specified command line parameters passed to 
ROMIO's configure script
  Complete set of command line parameters passed to 
ROMIO's configure script

openMPI4.1.0 + xpmem

$ mpirun --map-by core -rank-by core --bind-to core --mca pml ob1 --mca btl 
vader,self ./mpi-bench ic1

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-12 Thread Jeff Squyres (jsquyres) via users 
On Mar 12, 2021, at 1:20 AM, Raut, S Biplab via users 
mailto:users@lists.open-mpi.org>> wrote:

Reposting here without the logs – it seems there is a message size limit of 
150KB and so could not attach the logs.
(Request the moderator to approve the original mail that has attachment of 
compressed logs)

Sorry -- we can't approve messages more than 150KB because our web mail archive 
service (https://www.mail-archive.com/users@lists.open-mpi.org/) is the one 
imposing the 150KB limit.  If you have a big attachment, you might try a higher 
/ better level of compression (e.g., try bz2 or zst or something), or post your 
logs elsewhere and link to them in your email.

You can also try deleting  a lot of the quoted text in the email; all that HTML 
formatting also takes up quite a bit of space.

Sorry!

--
Jeff Squyres
jsquy...@cisco.com

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-08 Thread Raut, S Biplab via users 
Any suggestions or hints on the performance anomalies observed my me ?

By the way, it would be good to know if there is any mechanism/tool to do 
performance comparison between two openMPI versions (let’s say an older 3.1.1 
version vs a new stable release 4.1.0 version) ?

With Regards,
S. Biplab Raut

From: users  On Behalf Of Raut, S Biplab via 
users
Sent: Sunday, March 7, 2021 5:37 PM
To: Gilles Gouaillardet 
Cc: Raut, S Biplab ; Open MPI Users 

Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?

[CAUTION: External Email]
Dear Gilles,
Thank you. Please check my replies inline.

First you need to make sure your app is correctly pinned in order to measure 
optimal and stable performances.
If this is flat MPI,
mpirun --bind-to core ...
will do the trick.
On the single-node 128 cores, I use the below options to properly bind/rank the 
MPI processes.
mpirun --map-by core --rank-by core --bind-to core -np 128  


Best/not best is imho a pretty unusable metric.
You first have to check how stable the performances are (by running each app a 
few times), and then share normalized performance
(for example, best = 1, not best = 0.xy)
0.98 is ok-ish, 0.80 is not
The test bench simulation is done for more than 3000 iterations to minimize 
run-to-run variations, and the best/max GLOPS is selected out of all the runs.
This is a standalone benchmark that we usually run. Finally the evaluated 
source code is used in various HPC/scientific codes like NAMD, GROMACS, QE, 
VASP.
I am sorry that I could not provide the absolute performance numbers since I 
would need official approvals.
But, please be assured that the “best” vs “not-best” is actually qualified 
based on actual gap (atleast 10% or more) and not run-to-run variations.


With Regards,
S. Biplab Raut

From: Gilles Gouaillardet 
mailto:gilles.gouaillar...@gmail.com>>
Sent: Sunday, March 7, 2021 4:56 PM
To: Raut, S Biplab mailto:biplab.r...@amd.com>>
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?

[CAUTION: External Email]
Hi,


First you need to make sure your app is correctly pinned in order to measure 
optimal and stable performances.
If this is flat MPI,
mpirun --bind-to core ...
will do the trick.

Best/not best is imho a pretty unusable metric.
You first have to check how stable the performances are (by running each app a 
few times), and then share normalized performance
(for example, best = 1, not best = 0.xy)
0.98 is ok-ish, 0.80 is not

Cheers,

Gilles

On Sun, Mar 7, 2021 at 4:30 AM Raut, S Biplab 
mailto:biplab.r...@amd.com>> wrote:

Dear Gilles and Nathan,

  Thank you for your suggestions.

I have experimented with them and got few questions for you.



My application is open-source FFTW library and its MPI test bench - It makes 
use of distributed/MPI global transpose for a given MPI input problem.

I ran the program on 5 different combinations of OS and openMPI version/flags. 
Please check below few sample cases for which performance vary a lot across 
these 5 combinations.



Single-node 128 MPI ranks

GFLOPS comparison (absolute numbers not provided)



Double-precision complex type 1D array of size

Ubuntu 19.04 + openMPI3.1.1

Ubuntu 20.04 + openMPI4.1.0

Ubuntu 20.04 + openMPI4.1.0 +

Using --mca pml ob1 --mca btl vader,self,

Ubuntu 20.04 + openMPI4.1.0 +

xpmem

Ubuntu 20.04 + openMPI4.1.0 +
xpmem
+

Using --mca pml ob1 --mca btl vader,self,

Comments

390625

Best

Not-best

Not-best

Not-best

Not-best

Option with (Ubuntu 19.04 + openMPI3.1.1) is best

2097152

Not-best

Not-best

Not-best

Not-best

Best

Option with (Ubuntu 20.04 + openMPI4.1.0 + xpmem

+ Using --mca pml ob1 --mca btl vader,self,) is best

4194304

Not-best

Not-best

Not-best

Best

Not-best

Option with (Ubuntu 20.04 + openMPI4.1.0 + xpmem) is best

6400

Not-best

Best

Not-best

Not-best

Not-best

Option with (Ubuntu 20.04 + openMPI4.1.0) is best



My questions are:-

  1.  I was using openMPI3.1.1 on Ubuntu19.04 without “xpmem” and “runtime mca 
vader option”, then why the plain/stock openMPI4.1.0 on Ubuntu20.04 is not 
giving the best performance?
  2.  In most of the cases using “xpmem” library gives the best performance, 
but for few cases “Ubuntu 19.04 + openMPI3.1.1” is best. How to finalize which 
version to use universally?
  3.  I was getting a runtime warning for “xpmem” as mentioned below:-

WARNING: Could not generate an xpmem segment id for this process’ address space.

The vader shared memory BTL will fall back on another single-copy mechanish if 
one is available. This may result in lower performance.

How to resolve this issue?



With Regards,

S. Biplab Raut



-Original Message-
From: users 
mailto:users-boun...@lists.open-mpi.org>> On 
Behalf Of Gilles Gouaillardet via users
Sent: Friday, March 5, 2021 5:58 AM
To: Open MPI Users mailto:users@lists.open-mpi.org&g

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-07 Thread Raut, S Biplab via users 
Dear Gilles,
Thank you. Please check my replies inline.

First you need to make sure your app is correctly pinned in order to measure 
optimal and stable performances.
If this is flat MPI,
mpirun --bind-to core ...
will do the trick.
On the single-node 128 cores, I use the below options to properly bind/rank the 
MPI processes.
mpirun --map-by core --rank-by core --bind-to core -np 128  


Best/not best is imho a pretty unusable metric.
You first have to check how stable the performances are (by running each app a 
few times), and then share normalized performance
(for example, best = 1, not best = 0.xy)
0.98 is ok-ish, 0.80 is not
The test bench simulation is done for more than 3000 iterations to minimize 
run-to-run variations, and the best/max GLOPS is selected out of all the runs.
This is a standalone benchmark that we usually run. Finally the evaluated 
source code is used in various HPC/scientific codes like NAMD, GROMACS, QE, 
VASP.
I am sorry that I could not provide the absolute performance numbers since I 
would need official approvals.
But, please be assured that the “best” vs “not-best” is actually qualified 
based on actual gap (atleast 10% or more) and not run-to-run variations.


With Regards,
S. Biplab Raut

From: Gilles Gouaillardet 
Sent: Sunday, March 7, 2021 4:56 PM
To: Raut, S Biplab 
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?

[CAUTION: External Email]
Hi,


First you need to make sure your app is correctly pinned in order to measure 
optimal and stable performances.
If this is flat MPI,
mpirun --bind-to core ...
will do the trick.

Best/not best is imho a pretty unusable metric.
You first have to check how stable the performances are (by running each app a 
few times), and then share normalized performance
(for example, best = 1, not best = 0.xy)
0.98 is ok-ish, 0.80 is not

Cheers,

Gilles

On Sun, Mar 7, 2021 at 4:30 AM Raut, S Biplab 
mailto:biplab.r...@amd.com>> wrote:

Dear Gilles and Nathan,

  Thank you for your suggestions.

I have experimented with them and got few questions for you.



My application is open-source FFTW library and its MPI test bench - It makes 
use of distributed/MPI global transpose for a given MPI input problem.

I ran the program on 5 different combinations of OS and openMPI version/flags. 
Please check below few sample cases for which performance vary a lot across 
these 5 combinations.



Single-node 128 MPI ranks

GFLOPS comparison (absolute numbers not provided)



Double-precision complex type 1D array of size

Ubuntu 19.04 + openMPI3.1.1

Ubuntu 20.04 + openMPI4.1.0

Ubuntu 20.04 + openMPI4.1.0 +

Using --mca pml ob1 --mca btl vader,self,

Ubuntu 20.04 + openMPI4.1.0 +

xpmem

Ubuntu 20.04 + openMPI4.1.0 +
xpmem
+

Using --mca pml ob1 --mca btl vader,self,

Comments

390625

Best

Not-best

Not-best

Not-best

Not-best

Option with (Ubuntu 19.04 + openMPI3.1.1) is best

2097152

Not-best

Not-best

Not-best

Not-best

Best

Option with (Ubuntu 20.04 + openMPI4.1.0 + xpmem

+ Using --mca pml ob1 --mca btl vader,self,) is best

4194304

Not-best

Not-best

Not-best

Best

Not-best

Option with (Ubuntu 20.04 + openMPI4.1.0 + xpmem) is best

6400

Not-best

Best

Not-best

Not-best

Not-best

Option with (Ubuntu 20.04 + openMPI4.1.0) is best



My questions are:-

  1.  I was using openMPI3.1.1 on Ubuntu19.04 without “xpmem” and “runtime mca 
vader option”, then why the plain/stock openMPI4.1.0 on Ubuntu20.04 is not 
giving the best performance?
  2.  In most of the cases using “xpmem” library gives the best performance, 
but for few cases “Ubuntu 19.04 + openMPI3.1.1” is best. How to finalize which 
version to use universally?
  3.  I was getting a runtime warning for “xpmem” as mentioned below:-

WARNING: Could not generate an xpmem segment id for this process’ address space.

The vader shared memory BTL will fall back on another single-copy mechanish if 
one is available. This may result in lower performance.

How to resolve this issue?



With Regards,

S. Biplab Raut



-Original Message-
From: users 
mailto:users-boun...@lists.open-mpi.org>> On 
Behalf Of Gilles Gouaillardet via users
Sent: Friday, March 5, 2021 5:58 AM
To: Open MPI Users mailto:users@lists.open-mpi.org>>
Cc: Gilles Gouaillardet 
mailto:gilles.gouaillar...@gmail.com>>
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?



[CAUTION: External Email]



On top of XPMEM, try to also force btl/vader with mpirun --mca pml ob1 --mca 
btl vader,self, ...



On Fri, Mar 5, 2021 at 8:37 AM Nathan Hjelm via users 
mailto:users@lists.open-mpi.org>> wrote:

>

> I would run the v4.x series and install xpmem if you can 
> (https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fhjelmn%2Fxpmemdata=04%7C01%7CBiplab.Raut%40amd.com%7C34ff47831b764d6add4808d8df6de2e1%7C3dd8961fe4884e608e11a8

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-06 Thread Raut, S Biplab via users 
Dear Gilles and Nathan,

  Thank you for your suggestions.

I have experimented with them and got few questions for you.



My application is open-source FFTW library and its MPI test bench - It makes 
use of distributed/MPI global transpose for a given MPI input problem.

I ran the program on 5 different combinations of OS and openMPI version/flags. 
Please check below few sample cases for which performance vary a lot across 
these 5 combinations.



Single-node 128 MPI ranks

GFLOPS comparison (absolute numbers not provided)



Double-precision complex type 1D array of size

Ubuntu 19.04 + openMPI3.1.1

Ubuntu 20.04 + openMPI4.1.0

Ubuntu 20.04 + openMPI4.1.0 +

Using --mca pml ob1 --mca btl vader,self,

Ubuntu 20.04 + openMPI4.1.0 +

xpmem

Ubuntu 20.04 + openMPI4.1.0 +
xpmem
+

Using --mca pml ob1 --mca btl vader,self,

Comments

390625

Best

Not-best

Not-best

Not-best

Not-best

Option with (Ubuntu 19.04 + openMPI3.1.1) is best

2097152

Not-best

Not-best

Not-best

Not-best

Best

Option with (Ubuntu 20.04 + openMPI4.1.0 + xpmem

+ Using --mca pml ob1 --mca btl vader,self,) is best

4194304

Not-best

Not-best

Not-best

Best

Not-best

Option with (Ubuntu 20.04 + openMPI4.1.0 + xpmem) is best

6400

Not-best

Best

Not-best

Not-best

Not-best

Option with (Ubuntu 20.04 + openMPI4.1.0) is best



My questions are:-

  1.  I was using openMPI3.1.1 on Ubuntu19.04 without “xpmem” and “runtime mca 
vader option”, then why the plain/stock openMPI4.1.0 on Ubuntu20.04 is not 
giving the best performance?
  2.  In most of the cases using “xpmem” library gives the best performance, 
but for few cases “Ubuntu 19.04 + openMPI3.1.1” is best. How to finalize which 
version to use universally?
  3.  I was getting a runtime warning for “xpmem” as mentioned below:-

WARNING: Could not generate an xpmem segment id for this process’ address space.

The vader shared memory BTL will fall back on another single-copy mechanish if 
one is available. This may result in lower performance.

How to resolve this issue?



With Regards,

S. Biplab Raut



-Original Message-
From: users  On Behalf Of Gilles Gouaillardet 
via users
Sent: Friday, March 5, 2021 5:58 AM
To: Open MPI Users 
Cc: Gilles Gouaillardet 
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?



[CAUTION: External Email]



On top of XPMEM, try to also force btl/vader with mpirun --mca pml ob1 --mca 
btl vader,self, ...



On Fri, Mar 5, 2021 at 8:37 AM Nathan Hjelm via users 
mailto:users@lists.open-mpi.org>> wrote:

>

> I would run the v4.x series and install xpmem if you can 
> (https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fhjelmn%2Fxpmemdata=04%7C01%7CBiplab.Raut%40amd.com%7C34ff47831b764d6add4808d8df6de2e1%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637505010367426915%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=MOIgiiQ5J9qIaD7uBjCbPkuAOhioSxLiyDQbOJspGAU%3Dreserved=0).
>  You will need to build with —with-xpmem=/path/to/xpmem to use xpmem 
> otherwise vader will default to using CMA. This will provide the best 
> possible performance.

>

> -Nathan

>

> On Mar 4, 2021, at 5:55 AM, Raut, S Biplab via users 
> mailto:users@lists.open-mpi.org>> wrote:

>

> [AMD Official Use Only - Internal Distribution Only]

>

> It is a single node execution, so it should be using shared memory (vader).

>

> With Regards,

> S. Biplab Raut

>

> From: Heinz, Michael William

> mailto:michael.william.he...@cornelisnetworks.com>>

> Sent: Thursday, March 4, 2021 5:17 PM

> To: Open MPI Users mailto:users@lists.open-mpi.org>>

> Cc: Raut, S Biplab mailto:biplab.r...@amd.com>>

> Subject: Re: [OMPI users] Stable and performant openMPI version for 
> Ubuntu20.04 ?

>

> [CAUTION: External Email]

>

> What interconnect are you using at run time? That is, are you using Ethernet 
> or InfiniBand or Omnipath?

>

> Sent from my iPad

>

>

>

> On Mar 4, 2021, at 5:05 AM, Raut, S Biplab via users 
> mailto:users@lists.open-mpi.org>> wrote:

>

> 

> [AMD Official Use Only - Internal Distribution Only]

>

> After downloading a particular openMPI version, let’s say v3.1.1 from 
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdownload.open-mpi.org%2Frelease%2Fopen-mpi%2Fv3.1%2Fopenmpi-3.1.1.tar.gzdata=04%7C01%7CBiplab.Raut%40amd.com%7C34ff47831b764d6add4808d8df6de2e1%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637505010367426915%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=pc7oejptvf2zmmhpil4LepWcjDoIfTFFTgLEa%2Bld8CE%3Dreserved=0
>  , I follow the below steps.

> ./configure --prefix="$INSTALL_DIR" --enable-mpi-fortran --enable-m

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-04 Thread Mahesh Shinde via users 
Pl unsubscribe me.

On Fri, 5 Mar, 2021, 6:02 AM Gilles Gouaillardet via users, <
users@lists.open-mpi.org> wrote:

> On top of XPMEM, try to also force btl/vader with
> mpirun --mca pml ob1 --mca btl vader,self, ...
>
> On Fri, Mar 5, 2021 at 8:37 AM Nathan Hjelm via users
>  wrote:
> >
> > I would run the v4.x series and install xpmem if you can (
> http://github.com/hjelmn/xpmem). You will need to build with
> —with-xpmem=/path/to/xpmem to use xpmem otherwise vader will default to
> using CMA. This will provide the best possible performance.
> >
> > -Nathan
> >
> > On Mar 4, 2021, at 5:55 AM, Raut, S Biplab via users <
> users@lists.open-mpi.org> wrote:
> >
> > [AMD Official Use Only - Internal Distribution Only]
> >
> > It is a single node execution, so it should be using shared memory
> (vader).
> >
> > With Regards,
> > S. Biplab Raut
> >
> > From: Heinz, Michael William  >
> > Sent: Thursday, March 4, 2021 5:17 PM
> > To: Open MPI Users 
> > Cc: Raut, S Biplab 
> > Subject: Re: [OMPI users] Stable and performant openMPI version for
> Ubuntu20.04 ?
> >
> > [CAUTION: External Email]
> >
> > What interconnect are you using at run time? That is, are you using
> Ethernet or InfiniBand or Omnipath?
> >
> > Sent from my iPad
> >
> >
> >
> > On Mar 4, 2021, at 5:05 AM, Raut, S Biplab via users <
> users@lists.open-mpi.org> wrote:
> >
> > 
> > [AMD Official Use Only - Internal Distribution Only]
> >
> > After downloading a particular openMPI version, let’s say v3.1.1 from
> https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.1.tar.gz
> , I follow the below steps.
> > ./configure --prefix="$INSTALL_DIR" --enable-mpi-fortran
> --enable-mpi-cxx --enable-shared=yes --enable-static=yes
> --enable-mpi1-compatibility
> >   make -j
> >   make install
> >   export PATH=$INSTALL_DIR/bin:$PATH
> >   export LD_LIBRARY_PATH=$INSTALL_DIR/lib:$LD_LIBRARY_PATH
> > Additionally, I also install libnuma-dev on the machine.
> >
> > For all the machines having Ubuntu 18.04 and 19.04, it works correctly
> and results in expected performance/GFLOPS.
> > But, when OS is changed to Ubuntu 20.04, then I start getting the issues
> as mentioned in my original/previous mail below.
> >
> > With Regards,
> > S. Biplab Raut
> >
> > From: users  On Behalf Of John Hearns
> via users
> > Sent: Thursday, March 4, 2021 1:53 PM
> > To: Open MPI Users 
> > Cc: John Hearns 
> > Subject: Re: [OMPI users] Stable and performant openMPI version for
> Ubuntu20.04 ?
> >
> > [CAUTION: External Email]
> > How are you installing the OpenMPI versions? Are you using packages
> which are distributed by the OS?
> >
> > It might be worth looking at using Easybuid or Spack
> > https://docs.easybuild.io/en/latest/Introduction.html
> > https://spack.readthedocs.io/en/latest/
> >
> >
> > On Thu, 4 Mar 2021 at 07:35, Raut, S Biplab via users <
> users@lists.open-mpi.org> wrote:
> >
> > [AMD Official Use Only - Internal Distribution Only]
> >
> > Dear Experts,
> > Until recently, I was using openMPI3.1.1 to run
> single node 128 ranks MPI application on Ubuntu18.04 and Ubuntu19.04.
> > But, now the OS on these machines are upgraded to Ubuntu20.04, and I
> have been observing program hangs with openMPI3.1.1 version.
> > So, I tried with openMPI4.0.5 version – The program ran properly without
> any issues but there is a performance regression in my application.
> >
> > Can I know the stable openMPI version recommended for Ubuntu20.04 that
> has no known regression compared to v3.1.1.
> >
> > With Regards,
> > S. Biplab Raut
> >
> >
>

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-04 Thread Gilles Gouaillardet via users 
On top of XPMEM, try to also force btl/vader with
mpirun --mca pml ob1 --mca btl vader,self, ...

On Fri, Mar 5, 2021 at 8:37 AM Nathan Hjelm via users
 wrote:
>
> I would run the v4.x series and install xpmem if you can 
> (http://github.com/hjelmn/xpmem). You will need to build with 
> —with-xpmem=/path/to/xpmem to use xpmem otherwise vader will default to using 
> CMA. This will provide the best possible performance.
>
> -Nathan
>
> On Mar 4, 2021, at 5:55 AM, Raut, S Biplab via users 
>  wrote:
>
> [AMD Official Use Only - Internal Distribution Only]
>
> It is a single node execution, so it should be using shared memory (vader).
>
> With Regards,
> S. Biplab Raut
>
> From: Heinz, Michael William 
> Sent: Thursday, March 4, 2021 5:17 PM
> To: Open MPI Users 
> Cc: Raut, S Biplab 
> Subject: Re: [OMPI users] Stable and performant openMPI version for 
> Ubuntu20.04 ?
>
> [CAUTION: External Email]
>
> What interconnect are you using at run time? That is, are you using Ethernet 
> or InfiniBand or Omnipath?
>
> Sent from my iPad
>
>
>
> On Mar 4, 2021, at 5:05 AM, Raut, S Biplab via users 
>  wrote:
>
> 
> [AMD Official Use Only - Internal Distribution Only]
>
> After downloading a particular openMPI version, let’s say v3.1.1 from 
> https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.1.tar.gz , I 
> follow the below steps.
> ./configure --prefix="$INSTALL_DIR" --enable-mpi-fortran --enable-mpi-cxx 
> --enable-shared=yes --enable-static=yes --enable-mpi1-compatibility
>   make -j
>   make install
>   export PATH=$INSTALL_DIR/bin:$PATH
>   export LD_LIBRARY_PATH=$INSTALL_DIR/lib:$LD_LIBRARY_PATH
> Additionally, I also install libnuma-dev on the machine.
>
> For all the machines having Ubuntu 18.04 and 19.04, it works correctly and 
> results in expected performance/GFLOPS.
> But, when OS is changed to Ubuntu 20.04, then I start getting the issues as 
> mentioned in my original/previous mail below.
>
> With Regards,
> S. Biplab Raut
>
> From: users  On Behalf Of John Hearns via 
> users
> Sent: Thursday, March 4, 2021 1:53 PM
> To: Open MPI Users 
> Cc: John Hearns 
> Subject: Re: [OMPI users] Stable and performant openMPI version for 
> Ubuntu20.04 ?
>
> [CAUTION: External Email]
> How are you installing the OpenMPI versions? Are you using packages which are 
> distributed by the OS?
>
> It might be worth looking at using Easybuid or Spack
> https://docs.easybuild.io/en/latest/Introduction.html
> https://spack.readthedocs.io/en/latest/
>
>
> On Thu, 4 Mar 2021 at 07:35, Raut, S Biplab via users 
>  wrote:
>
> [AMD Official Use Only - Internal Distribution Only]
>
> Dear Experts,
> Until recently, I was using openMPI3.1.1 to run 
> single node 128 ranks MPI application on Ubuntu18.04 and Ubuntu19.04.
> But, now the OS on these machines are upgraded to Ubuntu20.04, and I have 
> been observing program hangs with openMPI3.1.1 version.
> So, I tried with openMPI4.0.5 version – The program ran properly without any 
> issues but there is a performance regression in my application.
>
> Can I know the stable openMPI version recommended for Ubuntu20.04 that has no 
> known regression compared to v3.1.1.
>
> With Regards,
> S. Biplab Raut
>
>

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-04 Thread Nathan Hjelm via users 
I would run the v4.x series and install xpmem if you can 
(http://github.com/hjelmn/xpmem <http://github.com/hjelmn/xpmem>). You will 
need to build with —with-xpmem=/path/to/xpmem to use xpmem otherwise vader will 
default to using CMA. This will provide the best possible performance.

-Nathan

> On Mar 4, 2021, at 5:55 AM, Raut, S Biplab via users 
>  wrote:
> 
> [AMD Official Use Only - Internal Distribution Only]
>  
> It is a single node execution, so it should be using shared memory (vader).
>  
> With Regards,
> S. Biplab Raut
>  
> From: Heinz, Michael William  <mailto:michael.william.he...@cornelisnetworks.com>> 
> Sent: Thursday, March 4, 2021 5:17 PM
> To: Open MPI Users  <mailto:users@lists.open-mpi.org>>
> Cc: Raut, S Biplab mailto:biplab.r...@amd.com>>
> Subject: Re: [OMPI users] Stable and performant openMPI version for 
> Ubuntu20.04 ?
>  
> [CAUTION: External Email] 
> What interconnect are you using at run time? That is, are you using Ethernet 
> or InfiniBand or Omnipath?
> 
> Sent from my iPad
>  
> 
> On Mar 4, 2021, at 5:05 AM, Raut, S Biplab via users 
> mailto:users@lists.open-mpi.org>> wrote:
> 
>  
> [AMD Official Use Only - Internal Distribution Only]
>  
> After downloading a particular openMPI version, let’s say v3.1.1 from 
> https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.1.tar.gz 
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdownload.open-mpi.org%2Frelease%2Fopen-mpi%2Fv3.1%2Fopenmpi-3.1.1.tar.gz=04%7C01%7CBiplab.Raut%40amd.com%7C9e9baf59cf66417e2e1808d8df0344e3%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637504552448339561%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=eeLrTeKE%2FwYobg0t0WN8TrGsbJ8xfjq6lJQgxptImUw%3D=0>
>  , I follow the below steps.
> ./configure --prefix="$INSTALL_DIR" --enable-mpi-fortran --enable-mpi-cxx 
> --enable-shared=yes --enable-static=yes --enable-mpi1-compatibility
>   make -j
>   make install
>   export PATH=$INSTALL_DIR/bin:$PATH
>   export LD_LIBRARY_PATH=$INSTALL_DIR/lib:$LD_LIBRARY_PATH
> Additionally, I also install libnuma-dev on the machine.
>  
> For all the machines having Ubuntu 18.04 and 19.04, it works correctly and 
> results in expected performance/GFLOPS.
> But, when OS is changed to Ubuntu 20.04, then I start getting the issues as 
> mentioned in my original/previous mail below.
>  
> With Regards,
> S. Biplab Raut
>  
> From: users  <mailto:users-boun...@lists.open-mpi.org>> On Behalf Of John Hearns via users
> Sent: Thursday, March 4, 2021 1:53 PM
> To: Open MPI Users  <mailto:users@lists.open-mpi.org>>
> Cc: John Hearns mailto:hear...@gmail.com>>
> Subject: Re: [OMPI users] Stable and performant openMPI version for 
> Ubuntu20.04 ?
>  
> [CAUTION: External Email] 
> How are you installing the OpenMPI versions? Are you using packages which are 
> distributed by the OS?
>  
> It might be worth looking at using Easybuid or Spack
> https://docs.easybuild.io/en/latest/Introduction.html 
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.easybuild.io%2Fen%2Flatest%2FIntroduction.html=04%7C01%7CBiplab.Raut%40amd.com%7C9e9baf59cf66417e2e1808d8df0344e3%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637504552448339561%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=qgr6YMUWZ184t3OE6UwRvE3lLdQLqAy26kr5ygoTctM%3D=0>
> https://spack.readthedocs.io/en/latest/ 
> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspack.readthedocs.io%2Fen%2Flatest%2F=04%7C01%7CBiplab.Raut%40amd.com%7C9e9baf59cf66417e2e1808d8df0344e3%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637504552448349558%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=XauzRNeAfMI1mpKrbhL7UeWs4UrTMtZjfgmX29BAN2Y%3D=0>
>  
>  
> On Thu, 4 Mar 2021 at 07:35, Raut, S Biplab via users 
> mailto:users@lists.open-mpi.org>> wrote:
> [AMD Official Use Only - Internal Distribution Only]
>  
> Dear Experts,
> Until recently, I was using openMPI3.1.1 to run 
> single node 128 ranks MPI application on Ubuntu18.04 and Ubuntu19.04.
> But, now the OS on these machines are upgraded to Ubuntu20.04, and I have 
> been observing program hangs with openMPI3.1.1 version.
> So, I tried with openMPI4.0.5 version – The program ran properly without any 
> issues but there is a performance regression in my application.
>  
> Can I know the stable openMPI version recommended for Ubuntu20.04 that has no 
> known regression compared to v3.1.1.
>  
> With Regards,
> S. Biplab Raut

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-04 Thread Raut, S Biplab via users 
Extremely sorry about that label – It is a default label applied to the emails.
Will make sure it does not get applied again.

With Regards,
S. Biplab Raut

From: users  On Behalf Of Ralph Castain via 
users
Sent: Thursday, March 4, 2021 7:11 PM
To: Open MPI Users 
Cc: Ralph Castain 
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?

[CAUTION: External Email]
Excuse me, but would you please ensure that you do not send mail to a mailing 
list containing this label:

[AMD Official Use Only - Internal Distribution Only]

Thank you
Ralph



On Mar 4, 2021, at 4:55 AM, Raut, S Biplab via users 
mailto:users@lists.open-mpi.org>> wrote:

[AMD Official Use Only - Internal Distribution Only]

It is a single node execution, so it should be using shared memory (vader).

With Regards,
S. Biplab Raut

From: Heinz, Michael William 
mailto:michael.william.he...@cornelisnetworks.com>>
Sent: Thursday, March 4, 2021 5:17 PM
To: Open MPI Users mailto:users@lists.open-mpi.org>>
Cc: Raut, S Biplab mailto:biplab.r...@amd.com>>
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?

[CAUTION: External Email]
What interconnect are you using at run time? That is, are you using Ethernet or 
InfiniBand or Omnipath?
Sent from my iPad

On Mar 4, 2021, at 5:05 AM, Raut, S Biplab via users 
mailto:users@lists.open-mpi.org>> wrote:

[AMD Official Use Only - Internal Distribution Only]

After downloading a particular openMPI version, let’s say v3.1.1 from 
https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.1.tar.gz<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdownload.open-mpi.org%2Frelease%2Fopen-mpi%2Fv3.1%2Fopenmpi-3.1.1.tar.gz=04%7C01%7CBiplab.Raut%40amd.com%7Cf49eb5c87fd74853a1e908d8df13932f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637504622460680102%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=DlTOFlxvXNbBKAb6J4SPoSaiXeGHhhR8g5aE3ULQ7ww%3D=0>,
 I follow the below steps.
./configure --prefix="$INSTALL_DIR" --enable-mpi-fortran --enable-mpi-cxx 
--enable-shared=yes --enable-static=yes --enable-mpi1-compatibility
  make -j
  make install
  export PATH=$INSTALL_DIR/bin:$PATH
  export LD_LIBRARY_PATH=$INSTALL_DIR/lib:$LD_LIBRARY_PATH
Additionally, I also install libnuma-dev on the machine.

For all the machines having Ubuntu 18.04 and 19.04, it works correctly and 
results in expected performance/GFLOPS.
But, when OS is changed to Ubuntu 20.04, then I start getting the issues as 
mentioned in my original/previous mail below.

With Regards,
S. Biplab Raut

From: users 
mailto:users-boun...@lists.open-mpi.org>> On 
Behalf Of John Hearns via users
Sent: Thursday, March 4, 2021 1:53 PM
To: Open MPI Users mailto:users@lists.open-mpi.org>>
Cc: John Hearns mailto:hear...@gmail.com>>
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?

[CAUTION: External Email]
How are you installing the OpenMPI versions? Are you using packages which are 
distributed by the OS?

It might be worth looking at using Easybuid or Spack
https://docs.easybuild.io/en/latest/Introduction.html<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.easybuild.io%2Fen%2Flatest%2FIntroduction.html=04%7C01%7CBiplab.Raut%40amd.com%7Cf49eb5c87fd74853a1e908d8df13932f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637504622460690098%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=zHIEUn2oGIF5JY%2BeltG7MgBZvs7Xal6KP6fdVclxCj8%3D=0>
https://spack.readthedocs.io/en/latest/<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspack.readthedocs.io%2Fen%2Flatest%2F=04%7C01%7CBiplab.Raut%40amd.com%7Cf49eb5c87fd74853a1e908d8df13932f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637504622460700089%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=IlPvmqcKu3WYxylVODs6S1HYc8OD%2F8XKmhEyXZbmeRs%3D=0>


On Thu, 4 Mar 2021 at 07:35, Raut, S Biplab via users 
mailto:users@lists.open-mpi.org>> wrote:
[AMD Official Use Only - Internal Distribution Only]

Dear Experts,
Until recently, I was using openMPI3.1.1 to run single 
node 128 ranks MPI application on Ubuntu18.04 and Ubuntu19.04.
But, now the OS on these machines are upgraded to Ubuntu20.04, and I have been 
observing program hangs with openMPI3.1.1 version.
So, I tried with openMPI4.0.5 version – The program ran properly without any 
issues but there is a performance regression in my application.

Can I know the stable openMPI version recommended for Ubuntu20.04 that has no 
known regression compared to v3.1.1.

With Regards,
S. Biplab Raut

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-04 Thread Ralph Castain via users 
Excuse me, but would you please ensure that you do not send mail to a mailing 
list containing this label:

[AMD Official Use Only - Internal Distribution Only]

Thank you
Ralph


On Mar 4, 2021, at 4:55 AM, Raut, S Biplab via users mailto:users@lists.open-mpi.org> > wrote:

[AMD Official Use Only - Internal Distribution Only]
 It is a single node execution, so it should be using shared memory (vader).
 With Regards,
S. Biplab Raut
 From: Heinz, Michael William mailto:michael.william.he...@cornelisnetworks.com> > 
Sent: Thursday, March 4, 2021 5:17 PM
To: Open MPI Users mailto:users@lists.open-mpi.org> >
Cc: Raut, S Biplab mailto:biplab.r...@amd.com> >
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?
 [CAUTION: External Email] 
What interconnect are you using at run time? That is, are you using Ethernet or 
InfiniBand or Omnipath?

Sent from my iPad
 
On Mar 4, 2021, at 5:05 AM, Raut, S Biplab via users mailto:users@lists.open-mpi.org> > wrote:

 
[AMD Official Use Only - Internal Distribution Only]
 After downloading a particular openMPI version, let’s say v3.1.1 from 
https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.1.tar.gz, I 
follow the below steps.
./configure --prefix="$INSTALL_DIR" --enable-mpi-fortran --enable-mpi-cxx 
--enable-shared=yes --enable-static=yes --enable-mpi1-compatibility
  make -j
  make install
  export PATH=$INSTALL_DIR/bin:$PATH
  export LD_LIBRARY_PATH=$INSTALL_DIR/lib:$LD_LIBRARY_PATH
Additionally, I also install libnuma-dev on the machine.
 For all the machines having Ubuntu 18.04 and 19.04, it works correctly and 
results in expected performance/GFLOPS.
But, when OS is changed to Ubuntu 20.04, then I start getting the issues as 
mentioned in my original/previous mail below.
 With Regards,
S. Biplab Raut
 From: users mailto:users-boun...@lists.open-mpi.org> > On Behalf Of John Hearns via users

Sent: Thursday, March 4, 2021 1:53 PM
To: Open MPI Users mailto:users@lists.open-mpi.org> >
Cc: John Hearns mailto:hear...@gmail.com> >
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?
 [CAUTION: External Email] 
How are you installing the OpenMPI versions? Are you using packages which are 
distributed by the OS?
 It might be worth looking at using Easybuid or Spack
https://docs.easybuild.io/en/latest/Introduction.html
https://spack.readthedocs.io/en/latest/
  On Thu, 4 Mar 2021 at 07:35, Raut, S Biplab via users 
mailto:users@lists.open-mpi.org> > wrote:
[AMD Official Use Only - Internal Distribution Only]
 Dear Experts,
    Until recently, I was using openMPI3.1.1 to run single 
node 128 ranks MPI application on Ubuntu18.04 and Ubuntu19.04.
But, now the OS on these machines are upgraded to Ubuntu20.04, and I have been 
observing program hangs with openMPI3.1.1 version.
So, I tried with openMPI4.0.5 version – The program ran properly without any 
issues but there is a performance regression in my application.
 Can I know the stable openMPI version recommended for Ubuntu20.04 that has no 
known regression compared to v3.1.1.
 With Regards,
S. Biplab Raut

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-04 Thread Raut, S Biplab via users 
[AMD Official Use Only - Internal Distribution Only]

It is a single node execution, so it should be using shared memory (vader).

With Regards,
S. Biplab Raut

From: Heinz, Michael William 
Sent: Thursday, March 4, 2021 5:17 PM
To: Open MPI Users 
Cc: Raut, S Biplab 
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?

[CAUTION: External Email]
What interconnect are you using at run time? That is, are you using Ethernet or 
InfiniBand or Omnipath?
Sent from my iPad

On Mar 4, 2021, at 5:05 AM, Raut, S Biplab via users 
mailto:users@lists.open-mpi.org>> wrote:


[AMD Official Use Only - Internal Distribution Only]

After downloading a particular openMPI version, let’s say v3.1.1 from 
https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.1.tar.gz<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdownload.open-mpi.org%2Frelease%2Fopen-mpi%2Fv3.1%2Fopenmpi-3.1.1.tar.gz=04%7C01%7CBiplab.Raut%40amd.com%7C9e9baf59cf66417e2e1808d8df0344e3%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637504552448339561%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=eeLrTeKE%2FwYobg0t0WN8TrGsbJ8xfjq6lJQgxptImUw%3D=0>
 , I follow the below steps.

./configure --prefix="$INSTALL_DIR" --enable-mpi-fortran --enable-mpi-cxx 
--enable-shared=yes --enable-static=yes --enable-mpi1-compatibility
  make -j
  make install
  export PATH=$INSTALL_DIR/bin:$PATH
  export LD_LIBRARY_PATH=$INSTALL_DIR/lib:$LD_LIBRARY_PATH
Additionally, I also install libnuma-dev on the machine.

For all the machines having Ubuntu 18.04 and 19.04, it works correctly and 
results in expected performance/GFLOPS.
But, when OS is changed to Ubuntu 20.04, then I start getting the issues as 
mentioned in my original/previous mail below.

With Regards,
S. Biplab Raut

From: users 
mailto:users-boun...@lists.open-mpi.org>> On 
Behalf Of John Hearns via users
Sent: Thursday, March 4, 2021 1:53 PM
To: Open MPI Users mailto:users@lists.open-mpi.org>>
Cc: John Hearns mailto:hear...@gmail.com>>
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?

[CAUTION: External Email]
How are you installing the OpenMPI versions? Are you using packages which are 
distributed by the OS?

It might be worth looking at using Easybuid or Spack
https://docs.easybuild.io/en/latest/Introduction.html<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.easybuild.io%2Fen%2Flatest%2FIntroduction.html=04%7C01%7CBiplab.Raut%40amd.com%7C9e9baf59cf66417e2e1808d8df0344e3%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637504552448339561%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=qgr6YMUWZ184t3OE6UwRvE3lLdQLqAy26kr5ygoTctM%3D=0>
https://spack.readthedocs.io/en/latest/<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspack.readthedocs.io%2Fen%2Flatest%2F=04%7C01%7CBiplab.Raut%40amd.com%7C9e9baf59cf66417e2e1808d8df0344e3%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637504552448349558%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=XauzRNeAfMI1mpKrbhL7UeWs4UrTMtZjfgmX29BAN2Y%3D=0>


On Thu, 4 Mar 2021 at 07:35, Raut, S Biplab via users 
mailto:users@lists.open-mpi.org>> wrote:

[AMD Official Use Only - Internal Distribution Only]

Dear Experts,
Until recently, I was using openMPI3.1.1 to run single 
node 128 ranks MPI application on Ubuntu18.04 and Ubuntu19.04.
But, now the OS on these machines are upgraded to Ubuntu20.04, and I have been 
observing program hangs with openMPI3.1.1 version.
So, I tried with openMPI4.0.5 version – The program ran properly without any 
issues but there is a performance regression in my application.

Can I know the stable openMPI version recommended for Ubuntu20.04 that has no 
known regression compared to v3.1.1.

With Regards,
S. Biplab Raut

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-04 Thread Heinz, Michael William via users 
What interconnect are you using at run time? That is, are you using Ethernet or 
InfiniBand or Omnipath?

Sent from my iPad

On Mar 4, 2021, at 5:05 AM, Raut, S Biplab via users  
wrote:



[AMD Official Use Only - Internal Distribution Only]

After downloading a particular openMPI version, let’s say v3.1.1 from 
https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.1.tar.gz<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdownload.open-mpi.org%2Frelease%2Fopen-mpi%2Fv3.1%2Fopenmpi-3.1.1.tar.gz=04%7C01%7Cmichael.william.heinz%40cornelisnetworks.com%7C8f8e82e9272f46b08a1608d8def4fbb9%7C4dbdb7da74ee4b458747ef5ce5ebe68a%7C0%7C0%7C637504491099833683%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=A7ia%2FTfeTEntAcNxJ%2FCaDLVoBVlHFcMkMojFM3iVwBo%3D=0>
 , I follow the below steps.

./configure --prefix="$INSTALL_DIR" --enable-mpi-fortran --enable-mpi-cxx 
--enable-shared=yes --enable-static=yes --enable-mpi1-compatibility
  make -j
  make install
  export PATH=$INSTALL_DIR/bin:$PATH
  export LD_LIBRARY_PATH=$INSTALL_DIR/lib:$LD_LIBRARY_PATH
Additionally, I also install libnuma-dev on the machine.

For all the machines having Ubuntu 18.04 and 19.04, it works correctly and 
results in expected performance/GFLOPS.
But, when OS is changed to Ubuntu 20.04, then I start getting the issues as 
mentioned in my original/previous mail below.

With Regards,
S. Biplab Raut

From: users  On Behalf Of John Hearns via 
users
Sent: Thursday, March 4, 2021 1:53 PM
To: Open MPI Users 
Cc: John Hearns 
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?

[CAUTION: External Email]
How are you installing the OpenMPI versions? Are you using packages which are 
distributed by the OS?

It might be worth looking at using Easybuid or Spack
https://docs.easybuild.io/en/latest/Introduction.html<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.easybuild.io%2Fen%2Flatest%2FIntroduction.html=04%7C01%7Cmichael.william.heinz%40cornelisnetworks.com%7C8f8e82e9272f46b08a1608d8def4fbb9%7C4dbdb7da74ee4b458747ef5ce5ebe68a%7C0%7C0%7C637504491099833683%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=XDC7EPzYb23xFekaxFmMWFPLVi%2FiIzyQpjnWRJP9enA%3D=0>
https://spack.readthedocs.io/en/latest/<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspack.readthedocs.io%2Fen%2Flatest%2F=04%7C01%7Cmichael.william.heinz%40cornelisnetworks.com%7C8f8e82e9272f46b08a1608d8def4fbb9%7C4dbdb7da74ee4b458747ef5ce5ebe68a%7C0%7C0%7C637504491099843640%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=Pxgm8BVqHiakTEh2q68slFcMDJxK1otfmTyWFUOMUEY%3D=0>


On Thu, 4 Mar 2021 at 07:35, Raut, S Biplab via users 
mailto:users@lists.open-mpi.org>> wrote:

[AMD Official Use Only - Internal Distribution Only]

Dear Experts,
Until recently, I was using openMPI3.1.1 to run single 
node 128 ranks MPI application on Ubuntu18.04 and Ubuntu19.04.
But, now the OS on these machines are upgraded to Ubuntu20.04, and I have been 
observing program hangs with openMPI3.1.1 version.
So, I tried with openMPI4.0.5 version – The program ran properly without any 
issues but there is a performance regression in my application.

Can I know the stable openMPI version recommended for Ubuntu20.04 that has no 
known regression compared to v3.1.1.

With Regards,
S. Biplab Raut

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-04 Thread Raut, S Biplab via users 
[AMD Official Use Only - Internal Distribution Only]

After downloading a particular openMPI version, let's say v3.1.1 from 
https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.1.tar.gz , I 
follow the below steps.

./configure --prefix="$INSTALL_DIR" --enable-mpi-fortran --enable-mpi-cxx 
--enable-shared=yes --enable-static=yes --enable-mpi1-compatibility
  make -j
  make install
  export PATH=$INSTALL_DIR/bin:$PATH
  export LD_LIBRARY_PATH=$INSTALL_DIR/lib:$LD_LIBRARY_PATH
Additionally, I also install libnuma-dev on the machine.

For all the machines having Ubuntu 18.04 and 19.04, it works correctly and 
results in expected performance/GFLOPS.
But, when OS is changed to Ubuntu 20.04, then I start getting the issues as 
mentioned in my original/previous mail below.

With Regards,
S. Biplab Raut

From: users  On Behalf Of John Hearns via 
users
Sent: Thursday, March 4, 2021 1:53 PM
To: Open MPI Users 
Cc: John Hearns 
Subject: Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 
?

[CAUTION: External Email]
How are you installing the OpenMPI versions? Are you using packages which are 
distributed by the OS?

It might be worth looking at using Easybuid or Spack
https://docs.easybuild.io/en/latest/Introduction.html<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.easybuild.io%2Fen%2Flatest%2FIntroduction.html=04%7C01%7CBiplab.Raut%40amd.com%7Ce41f6b6855304803bd2e08d8dee71934%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637504431443228336%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=dWFeX%2F5%2ByjM2jP%2FkkpY5BBMAtxTRAYmC1Y4N7x7cwkI%3D=0>
https://spack.readthedocs.io/en/latest/<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fspack.readthedocs.io%2Fen%2Flatest%2F=04%7C01%7CBiplab.Raut%40amd.com%7Ce41f6b6855304803bd2e08d8dee71934%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637504431443238327%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=2IHJOyMgRxibyqNNOG7sSob%2FNucGcWlkOH47mnTfdA4%3D=0>


On Thu, 4 Mar 2021 at 07:35, Raut, S Biplab via users 
mailto:users@lists.open-mpi.org>> wrote:

[AMD Official Use Only - Internal Distribution Only]

Dear Experts,
Until recently, I was using openMPI3.1.1 to run single 
node 128 ranks MPI application on Ubuntu18.04 and Ubuntu19.04.
But, now the OS on these machines are upgraded to Ubuntu20.04, and I have been 
observing program hangs with openMPI3.1.1 version.
So, I tried with openMPI4.0.5 version - The program ran properly without any 
issues but there is a performance regression in my application.

Can I know the stable openMPI version recommended for Ubuntu20.04 that has no 
known regression compared to v3.1.1.

With Regards,
S. Biplab Raut

Re: [OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-04 Thread John Hearns via users 
How are you installing the OpenMPI versions? Are you using packages which
are distributed by the OS?

It might be worth looking at using Easybuid or Spack
https://docs.easybuild.io/en/latest/Introduction.html
https://spack.readthedocs.io/en/latest/


On Thu, 4 Mar 2021 at 07:35, Raut, S Biplab via users <
users@lists.open-mpi.org> wrote:

> [AMD Official Use Only - Internal Distribution Only]
>
> Dear Experts,
>
> Until recently, I was using openMPI3.1.1 to run
> single node 128 ranks MPI application on Ubuntu18.04 and Ubuntu19.04.
>
> But, now the OS on these machines are upgraded to Ubuntu20.04, and I have
> been observing program hangs with openMPI3.1.1 version.
>
> So, I tried with openMPI4.0.5 version – The program ran properly without
> any issues but there is a performance regression in my application.
>
>
>
> Can I know the stable openMPI version recommended for Ubuntu20.04 that has
> no known regression compared to v3.1.1.
>
>
>
> With Regards,
>
> S. Biplab Raut
>

[OMPI users] Stable and performant openMPI version for Ubuntu20.04 ?

2021-03-03 Thread Raut, S Biplab via users 
[AMD Official Use Only - Internal Distribution Only]

Dear Experts,
Until recently, I was using openMPI3.1.1 to run single 
node 128 ranks MPI application on Ubuntu18.04 and Ubuntu19.04.
But, now the OS on these machines are upgraded to Ubuntu20.04, and I have been 
observing program hangs with openMPI3.1.1 version.
So, I tried with openMPI4.0.5 version - The program ran properly without any 
issues but there is a performance regression in my application.

Can I know the stable openMPI version recommended for Ubuntu20.04 that has no 
known regression compared to v3.1.1.

With Regards,
S. Biplab Raut