On 30/01/21 21:01, Walter Sklenka wrote:
Hi Giovanni!
Thats great! Many thanks for your fast and detailed answer!!!!
So this is the way we will go too!
Have a nice weekend and keep healthy!
Best regards
Walter
I suppose you will implement the solution with more recent versions of
the software components, so please let me know if everything works!
If yu have any issues I am ready to discuss!
Regards
Giovanni
-----Original Message-----
From: Giovanni Bracco <[email protected]>
Sent: Samstag, 30. Jänner 2021 18:08
To: gpfsug main discussion list <[email protected]>; Walter Sklenka
<[email protected]>
Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with
only ib rdma enabled
In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of
them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu
it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes
Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely
using RDMA since 2018. GPFS 4.2.3-19.
See
F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a multifabric GPFS
Spectrum Scale layout," 2019 International Conference on High Performance Computing
& Simulation (HPCS), Dublin, Ireland, 2019, pp. 1051-1052, doi:
10.1109/HPCS48598.2019.918813
When setting up the system the main trick has been:
just use CentOS drivers and do not install OFED We do not use IPoIB.
Giovanni
On 30/01/21 06:45, Walter Sklenka wrote:
Hi!
Is it possible to mix OPAcards and Infininiband HCAs on the same server?
In the faq
https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.
html#rdma
They talk about RDMA :
"RDMA is NOT supported on a node when both Mellanox HCAs and Intel
Omni-Path HFIs are ENABLED for RDMA."
So do I understand right: When we do NOT enable the opa interface we
can still enable IB ?
The reason I ask is, that we have a gpfs cluster of 6 NSD Servers
(wih access to storage) with opa interfaces which provide access to
remote cluster also via OPA.
A new cluster with HDR interfaces will be implemented soon
They shell have access to the same filesystems
When we add HDR interfaces to NSD servers and enable rdma on this
network while disabling rdma on opa we would accept the worse
performance via opa . We hope that this provides still better perf
and less technical overhead than using routers
Or am I totally wrong?
Thank you very much and keep healthy!
Best regards
Walter
Mit freundlichen Grüßen
*/Walter Sklenka/*
*/Technical Consultant/*
EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210
Wien
Tel: +43 1 29 22 165-31
Fax: +43 1 29 22 165-90
E-Mail: [email protected] <mailto:[email protected]>
Internet: www.edv-design.at <http://www.edv-design.at/>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
--
Giovanni Bracco
phone +39 351 8804788
E-mail [email protected]
WWW http://www.afs.enea.it/bracco
--
Giovanni Bracco
phone +39 351 8804788
E-mail [email protected]
WWW http://www.afs.enea.it/bracco
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
