Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled

[Walter Sklenka]

Hi Givanni !

I understand  and am convinced that the is an excellent solution  !!
Thank you very much!

-Original Message-
From: Giovanni Bracco  
Sent: Mittwoch, 3. Februar 2021 09:59
To: Walter Sklenka 
Cc: gpfsug main discussion list 
Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with 
only ib rdma enabled

We did not explore the issue of the IBM support and for budget limitation and 
for the mandatory integration of the data space between the two clusters, we 
decided to try the setup of the multi-fabric infrastructure and up to now it 
has been working without problems.

Giovanni

On 02/02/21 14:10, Walter Sklenka wrote:
> Hi Giovanni!
> 
> Thank you for your offer! 
> 
> it is planned to be implemented in June or so
> 
> We will use RHEL 8.x and newest gpfs version available
> 
> Only one question for this moment if I am allowed:
> 
> Did you ever ran into any problems with IBM support? I mean they say 
> in the FAQ shortly "not supported" , but do they in your environment 
> or do you accept that rdma problems would be needed to be fixed 
> without IBM
> 
> Thank you very much and have great days! And keep healthy!
> 
> Best regards walter
> 
> -Original Message-
> From: Giovanni Bracco 
> Sent: Montag, 1. Februar 2021 20:42
> To: Walter Sklenka 
> Cc: gpfsug main discussion list 
> Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD 
> Server with only ib rdma enabled
> 
> On 30/01/21 21:01, Walter Sklenka wrote:
> 
>  > Hi Giovanni!
> 
>  > Thats great! Many thanks for your fast and detailed answer
> 
>  > So this is the way we will go too!
> 
>  >
> 
>  > Have a nice weekend and keep healthy!
> 
>  > Best regards
> 
>  > Walter
> 
>  >
> 
> I suppose you will implement the solution with more recent versions of 
> the software components, so please let me know if everything works!
> 
> If yu have any issues I am ready to discuss!
> 
> Regards
> 
> Giovanni
> 
>  > -Original Message-
> 
>  > From: Giovanni Bracco  >
> 
>  > Sent: Samstag, 30. Jänner 2021 18:08
> 
>  > To: gpfsug main discussion list  >;
> 
>  > Walter Sklenka  >
> 
>  > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD
> 
>  > Server with only ib rdma enabled
> 
>  >
> 
>  > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, 
> each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes 
> SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main 
> OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to 
> DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19.
> 
>  > See
> 
>  > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of 
> a
> 
>  > multifabric GPFS Spectrum Scale layout," 2019 International 
> Conference
> 
>  > on High Performance Computing & Simulation (HPCS), Dublin, Ireland,
> 
>  > 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813
> 
>  >
> 
>  > When setting up the system the main trick has been:
> 
>  > just use CentOS drivers and do not install OFED We do not use IPoIB.
> 
>  >
> 
>  > Giovanni
> 
>  >
> 
>  > On 30/01/21 06:45, Walter Sklenka wrote:
> 
>  >> Hi!
> 
>  >>
> 
>  >> Is it possible to mix OPAcards and Infininiband HCAs on the same server?
> 
>  >>
> 
>  >> In the faq
> 
>  >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.
> 
>  >> html#rdma
> 
>  >>
> 
>  >>
> 
>  >> They talk about RDMA :
> 
>  >>
> 
>  >> "RDMA is NOT  supported on a node when both Mellanox HCAs and 
> Intel
> 
>  >> Omni-Path HFIs are ENABLED for RDMA."
> 
>  >>
> 
>  >> So do I understand right: When we do NOT enable  the opa interface 
> we
> 
>  >> can still enable IB ?
> 
>  >>
> 
>  >> The reason I ask  is, that we have a gpfs cluster of 6 NSD Servers
> 
>  >> (wih access to storage)  with opa interfaces which provide access 
> to
> 
>  >> remote cluster  also via OPA.
> 
>  >>
> 
>  >> A new cluster with HDR interfaces will be implemented soon
> 
>  >>
> 
>  >> They shell have access to the same filesystems
> 
>  >>
> 
>  >> When we add HDR interfaces to  NSD servers  and enable rdma on 
> this
> 
>  >> network  while disabling rdma on opa we would accept the worse
> 
>  >> performance via opa . We hope that this provides  still better 
> perf
> 
>  >> and less technical overhead  than using routers
> 
>  >>
> 
>  >> Or am I totally wrong?
> 
>  >>
> 
>  >> Thank you very much and keep healthy!
> 
>  >>
> 
>  >> Best regards
> 
>  >>
> 
>  >> Walter
> 
>  >>
> 
>  >> Mit freundlichen Grüßen
> 
>  >> */Walter Sklenka/*
> 
>  >> */Technical Consultant/*
> 
>  >>
> 
>  >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210
> 
>  >> Wien
> 
>  >> Tel: +43 1 29 22 165-31
> 
>  >> Fax: +43 1 29 22 165-90
> 
>  >> E-Mail: skle...@edv-design.at  
> 

Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled

[Giovanni Bracco]

We did not explore the issue of the IBM support and for budget 
limitation and for the mandatory integration of the data space between 
the two clusters, we decided to try the setup of the multi-fabric 
infrastructure and up to now it has been working without problems.


Giovanni

On 02/02/21 14:10, Walter Sklenka wrote:

Hi Giovanni!

Thank you for your offer! 

it is planned to be implemented in June or so

We will use RHEL 8.x and newest gpfs version available

Only one question for this moment if I am allowed:

Did you ever ran into any problems with IBM support? I mean they say in 
the FAQ shortly "not supported" , but do they in your environment or do 
you accept that rdma problems would be needed to be fixed without IBM


Thank you very much and have great days! And keep healthy!

Best regards walter

-Original Message-
From: Giovanni Bracco 
Sent: Montag, 1. Februar 2021 20:42
To: Walter Sklenka 
Cc: gpfsug main discussion list 
Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD 
Server with only ib rdma enabled


On 30/01/21 21:01, Walter Sklenka wrote:

 > Hi Giovanni!

 > Thats great! Many thanks for your fast and detailed answer

 > So this is the way we will go too!

 >

 > Have a nice weekend and keep healthy!

 > Best regards

 > Walter

 >

I suppose you will implement the solution with more recent versions of 
the software components, so please let me know if everything works!


If yu have any issues I am ready to discuss!

Regards

Giovanni

 > -Original Message-

 > From: Giovanni Bracco >


 > Sent: Samstag, 30. Jänner 2021 18:08

 > To: gpfsug main discussion list >;


 > Walter Sklenka >


 > Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD

 > Server with only ib rdma enabled

 >

 > In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, 
each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes 
SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA 
Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN 
storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19.


 > See

 > F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a

 > multifabric GPFS Spectrum Scale layout," 2019 International Conference

 > on High Performance Computing & Simulation (HPCS), Dublin, Ireland,

 > 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813

 >

 > When setting up the system the main trick has been:

 > just use CentOS drivers and do not install OFED We do not use IPoIB.

 >

 > Giovanni

 >

 > On 30/01/21 06:45, Walter Sklenka wrote:

 >> Hi!

 >>

 >> Is it possible to mix OPAcards and Infininiband HCAs on the same server?

 >>

 >> In the faq

 >> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.

 >> html#rdma

 >>

 >>

 >> They talk about RDMA :

 >>

 >> "RDMA is NOT  supported on a node when both Mellanox HCAs and Intel

 >> Omni-Path HFIs are ENABLED for RDMA."

 >>

 >> So do I understand right: When we do NOT enable  the opa interface we

 >> can still enable IB ?

 >>

 >> The reason I ask  is, that we have a gpfs cluster of 6 NSD Servers

 >> (wih access to storage)  with opa interfaces which provide access to

 >> remote cluster  also via OPA.

 >>

 >> A new cluster with HDR interfaces will be implemented soon

 >>

 >> They shell have access to the same filesystems

 >>

 >> When we add HDR interfaces to  NSD servers  and enable rdma on this

 >> network  while disabling rdma on opa we would accept the worse

 >> performance via opa . We hope that this provides  still better perf

 >> and less technical overhead  than using routers

 >>

 >> Or am I totally wrong?

 >>

 >> Thank you very much and keep healthy!

 >>

 >> Best regards

 >>

 >> Walter

 >>

 >> Mit freundlichen Grüßen

 >> */Walter Sklenka/*

 >> */Technical Consultant/*

 >>

 >> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210

 >> Wien

 >> Tel: +43 1 29 22 165-31

 >> Fax: +43 1 29 22 165-90

 >> E-Mail: skle...@edv-design.at  



 >> Internet: www.edv-design.at  



 >>

 >>

 >> ___

 >> gpfsug-discuss mailing list

 >> gpfsug-discuss at spectrumscale.org

 >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

 >>

 >

 > --

 > Giovanni Bracco

 > phone  +39 351 8804788

 > E-mail giovanni.bra...@enea.it 

 > WWW http://www.afs.enea.it/bracco

 >

--

Giovanni Bracco

phone  +39 351 8804788

E-mail giovanni.bra...@enea.it 

WWW http://www.afs.enea.it/bracco



--
Giovanni Bracco
phone  +39 351 8804788
E-mail  giovanni.bra...@enea.it
WWW http://www.afs.enea.it/bracco

Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled

[Walter Sklenka]

Hi Giovanni!

Thank you very much for your offer , we really would be very  grateful to be 
allowed to come if we run into troubles!

Well, the implementation will not happen before June or later, but may I ask 
only one question meanwhile?



Did you ever run into problems with IBM support or did you get a  special “OK”  
from them? Or do you accept to sove any rdma specific problems without support 
? (it´s only because of the FAQ “not supported” )



Have a great day and keep healthy!

Best regards walter





-Original Message-
From: Giovanni Bracco 
Sent: Montag, 1. Februar 2021 20:42
To: Walter Sklenka 
Cc: gpfsug main discussion list 
Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with 
only ib rdma enabled



On 30/01/21 21:01, Walter Sklenka wrote:

> Hi Giovanni!

> Thats great! Many thanks for your fast and detailed answer

> So this is the way we will go too!

>

> Have a nice weekend and keep healthy!

> Best regards

> Walter

>



I suppose you will implement the solution with more recent versions of the 
software components, so please let me know if everything works!



If yu have any issues I am ready to discuss!



Regards



Giovanni





> -Original Message-

> From: Giovanni Bracco 
> mailto:giovanni.bra...@enea.it>>

> Sent: Samstag, 30. Jänner 2021 18:08

> To: gpfsug main discussion list 
> mailto:gpfsug-discuss@spectrumscale.org>>;

> Walter Sklenka 
> mailto:walter.skle...@edv-design.at>>

> Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD

> Server with only ib rdma enabled

>

> In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of 
> them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge 
> cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 
> nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it 
> works nicely using RDMA since 2018. GPFS 4.2.3-19.

> See

> F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a

> multifabric GPFS Spectrum Scale layout," 2019 International Conference

> on High Performance Computing & Simulation (HPCS), Dublin, Ireland,

> 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813

>

> When setting up the system the main trick has been:

> just use CentOS drivers and do not install OFED We do not use IPoIB.

>

> Giovanni

>

> On 30/01/21 06:45, Walter Sklenka wrote:

>> Hi!

>>

>> Is it possible to mix OPAcards and Infininiband HCAs on the same server?

>>

>> In the faq

>> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.

>> html#rdma

>>

>>

>> They talk about RDMA :

>>

>> "RDMA is NOT  supported on a node when both Mellanox HCAs and Intel

>> Omni-Path HFIs are ENABLED for RDMA."

>>

>> So do I understand right: When we do NOT enable  the opa interface we

>> can still enable IB ?

>>

>> The reason I ask  is, that we have a gpfs cluster of 6 NSD Servers

>> (wih access to storage)  with opa interfaces which provide access to

>> remote cluster  also via OPA.

>>

>> A new cluster with HDR interfaces will be implemented soon

>>

>> They shell have access to the same filesystems

>>

>> When we add HDR interfaces to  NSD servers  and enable rdma on this

>> network  while disabling rdma on opa we would accept the worse

>> performance via opa . We hope that this provides  still better perf

>> and less technical overhead  than using routers

>>

>> Or am I totally wrong?

>>

>> Thank you very much and keep healthy!

>>

>> Best regards

>>

>> Walter

>>

>> Mit freundlichen Grüßen

>> */Walter Sklenka/*

>> */Technical Consultant/*

>>

>> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210

>> Wien

>> Tel: +43 1 29 22 165-31

>> Fax: +43 1 29 22 165-90

>> E-Mail: skle...@edv-design.at 
>> 

>> Internet: www.edv-design.at 
>> 

>>

>>

>> ___

>> gpfsug-discuss mailing list

>> gpfsug-discuss at spectrumscale.org

>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>>

>

> --

> Giovanni Bracco

> phone  +39 351 8804788

> E-mail  giovanni.bra...@enea.it

> WWW http://www.afs.enea.it/bracco

>



--

Giovanni Bracco

phone  +39 351 8804788

E-mail  giovanni.bra...@enea.it

WWW http://www.afs.enea.it/bracco
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled

[Walter Sklenka]

Hi Giovanni!

Thank you for your offer!  

it is planned to be implemented in June or so

We will use RHEL 8.x and newest gpfs version available



Only one question for this moment if I am allowed:

Did you ever ran into any problems with IBM support? I mean they say in the FAQ 
shortly "not supported" , but do they in your environment or do you accept that 
rdma problems would be needed to be fixed without IBM



Thank you very much and have great days! And keep healthy!

Best regards walter



-Original Message-
From: Giovanni Bracco 
Sent: Montag, 1. Februar 2021 20:42
To: Walter Sklenka 
Cc: gpfsug main discussion list 
Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with 
only ib rdma enabled



On 30/01/21 21:01, Walter Sklenka wrote:

> Hi Giovanni!

> Thats great! Many thanks for your fast and detailed answer

> So this is the way we will go too!

>

> Have a nice weekend and keep healthy!

> Best regards

> Walter

>



I suppose you will implement the solution with more recent versions of the 
software components, so please let me know if everything works!



If yu have any issues I am ready to discuss!



Regards



Giovanni





> -Original Message-

> From: Giovanni Bracco 
> mailto:giovanni.bra...@enea.it>>

> Sent: Samstag, 30. Jänner 2021 18:08

> To: gpfsug main discussion list 
> mailto:gpfsug-discuss@spectrumscale.org>>;

> Walter Sklenka 
> mailto:walter.skle...@edv-design.at>>

> Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD

> Server with only ib rdma enabled

>

> In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of 
> them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge 
> cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 
> nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it 
> works nicely using RDMA since 2018. GPFS 4.2.3-19.

> See

> F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a

> multifabric GPFS Spectrum Scale layout," 2019 International Conference

> on High Performance Computing & Simulation (HPCS), Dublin, Ireland,

> 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813

>

> When setting up the system the main trick has been:

> just use CentOS drivers and do not install OFED We do not use IPoIB.

>

> Giovanni

>

> On 30/01/21 06:45, Walter Sklenka wrote:

>> Hi!

>>

>> Is it possible to mix OPAcards and Infininiband HCAs on the same server?

>>

>> In the faq

>> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.

>> html#rdma

>>

>>

>> They talk about RDMA :

>>

>> "RDMA is NOT  supported on a node when both Mellanox HCAs and Intel

>> Omni-Path HFIs are ENABLED for RDMA."

>>

>> So do I understand right: When we do NOT enable  the opa interface we

>> can still enable IB ?

>>

>> The reason I ask  is, that we have a gpfs cluster of 6 NSD Servers

>> (wih access to storage)  with opa interfaces which provide access to

>> remote cluster  also via OPA.

>>

>> A new cluster with HDR interfaces will be implemented soon

>>

>> They shell have access to the same filesystems

>>

>> When we add HDR interfaces to  NSD servers  and enable rdma on this

>> network  while disabling rdma on opa we would accept the worse

>> performance via opa . We hope that this provides  still better perf

>> and less technical overhead  than using routers

>>

>> Or am I totally wrong?

>>

>> Thank you very much and keep healthy!

>>

>> Best regards

>>

>> Walter

>>

>> Mit freundlichen Grüßen

>> */Walter Sklenka/*

>> */Technical Consultant/*

>>

>> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210

>> Wien

>> Tel: +43 1 29 22 165-31

>> Fax: +43 1 29 22 165-90

>> E-Mail: skle...@edv-design.at 
>> 

>> Internet: www.edv-design.at 
>> 

>>

>>

>> ___

>> gpfsug-discuss mailing list

>> gpfsug-discuss at spectrumscale.org

>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

>>

>

> --

> Giovanni Bracco

> phone  +39 351 8804788

> E-mail  giovanni.bra...@enea.it

> WWW http://www.afs.enea.it/bracco

>



--

Giovanni Bracco

phone  +39 351 8804788

E-mail  giovanni.bra...@enea.it

WWW http://www.afs.enea.it/bracco
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled

[Giovanni Bracco]

On 30/01/21 21:01, Walter Sklenka wrote:

Hi Giovanni!
Thats great! Many thanks for your fast and detailed answer
So this is the way we will go too!

Have a nice weekend and keep healthy!
Best regards
Walter



I suppose you will implement the solution with more recent versions of 
the software components, so please let me know if everything works!


If yu have any issues I am ready to discuss!

Regards

Giovanni



-Original Message-
From: Giovanni Bracco 
Sent: Samstag, 30. Jänner 2021 18:08
To: gpfsug main discussion list ; Walter Sklenka 

Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with 
only ib rdma enabled

In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of 
them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu 
it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes 
Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely 
using RDMA since 2018. GPFS 4.2.3-19.
See
F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a multifabric GPFS 
Spectrum Scale layout," 2019 International Conference on High Performance Computing 
& Simulation (HPCS), Dublin, Ireland, 2019, pp. 1051-1052, doi: 
10.1109/HPCS48598.2019.918813

When setting up the system the main trick has been:
just use CentOS drivers and do not install OFED We do not use IPoIB.

Giovanni

On 30/01/21 06:45, Walter Sklenka wrote:

Hi!

Is it possible to mix OPAcards and Infininiband HCAs on the same server?

In the faq
https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.
html#rdma


They talk about RDMA :

"RDMA is NOT  supported on a node when both Mellanox HCAs and Intel
Omni-Path HFIs are ENABLED for RDMA."

So do I understand right: When we do NOT enable  the opa interface we
can still enable IB ?

The reason I ask  is, that we have a gpfs cluster of 6 NSD Servers
(wih access to storage)  with opa interfaces which provide access to
remote cluster  also via OPA.

A new cluster with HDR interfaces will be implemented soon

They shell have access to the same filesystems

When we add HDR interfaces to  NSD servers  and enable rdma on this
network  while disabling rdma on opa we would accept the worse
performance via opa . We hope that this provides  still better perf
and less technical overhead  than using routers

Or am I totally wrong?

Thank you very much and keep healthy!

Best regards

Walter

Mit freundlichen Grüßen
*/Walter Sklenka/*
*/Technical Consultant/*

EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210
Wien
Tel: +43 1 29 22 165-31
Fax: +43 1 29 22 165-90
E-Mail: skle...@edv-design.at 
Internet: www.edv-design.at 


___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



--
Giovanni Bracco
phone  +39 351 8804788
E-mail  giovanni.bra...@enea.it
WWW http://www.afs.enea.it/bracco



--
Giovanni Bracco
phone  +39 351 8804788
E-mail  giovanni.bra...@enea.it
WWW http://www.afs.enea.it/bracco
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled

[Walter Sklenka]

Hi Giovanni!
Thats great! Many thanks for your fast and detailed answer
So this is the way we will go too!

Have a nice weekend and keep healthy!
Best regards
Walter 

-Original Message-
From: Giovanni Bracco  
Sent: Samstag, 30. Jänner 2021 18:08
To: gpfsug main discussion list ; Walter 
Sklenka 
Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with 
only ib rdma enabled

In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of 
them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu 
it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes 
Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely 
using RDMA since 2018. GPFS 4.2.3-19.
See
F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a 
multifabric GPFS Spectrum Scale layout," 2019 International Conference on High 
Performance Computing & Simulation (HPCS), Dublin, Ireland, 2019, pp. 
1051-1052, doi: 10.1109/HPCS48598.2019.918813

When setting up the system the main trick has been:
just use CentOS drivers and do not install OFED We do not use IPoIB.

Giovanni

On 30/01/21 06:45, Walter Sklenka wrote:
> Hi!
> 
> Is it possible to mix OPAcards and Infininiband HCAs on the same server?
> 
> In the faq
> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.
> html#rdma
> 
> 
> They talk about RDMA :
> 
> "RDMA is NOT  supported on a node when both Mellanox HCAs and Intel 
> Omni-Path HFIs are ENABLED for RDMA."
> 
> So do I understand right: When we do NOT enable  the opa interface we 
> can still enable IB ?
> 
> The reason I ask  is, that we have a gpfs cluster of 6 NSD Servers  
> (wih access to storage)  with opa interfaces which provide access to 
> remote cluster  also via OPA.
> 
> A new cluster with HDR interfaces will be implemented soon
> 
> They shell have access to the same filesystems
> 
> When we add HDR interfaces to  NSD servers  and enable rdma on this 
> network  while disabling rdma on opa we would accept the worse 
> performance via opa . We hope that this provides  still better perf 
> and less technical overhead  than using routers
> 
> Or am I totally wrong?
> 
> Thank you very much and keep healthy!
> 
> Best regards
> 
> Walter
> 
> Mit freundlichen Grüßen
> */Walter Sklenka/*
> */Technical Consultant/*
> 
> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 
> Wien
> Tel: +43 1 29 22 165-31
> Fax: +43 1 29 22 165-90
> E-Mail: skle...@edv-design.at 
> Internet: www.edv-design.at 
> 
> 
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 

--
Giovanni Bracco
phone  +39 351 8804788
E-mail  giovanni.bra...@enea.it
WWW http://www.afs.enea.it/bracco
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss