Hi Giovanni! Thats great! Many thanks for your fast and detailed answer!!!! So this is the way we will go too!
Have a nice weekend and keep healthy! Best regards Walter -----Original Message----- From: Giovanni Bracco <giovanni.bra...@enea.it> Sent: Samstag, 30. Jänner 2021 18:08 To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>; Walter Sklenka <walter.skle...@edv-design.at> Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19. See F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a multifabric GPFS Spectrum Scale layout," 2019 International Conference on High Performance Computing & Simulation (HPCS), Dublin, Ireland, 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813 When setting up the system the main trick has been: just use CentOS drivers and do not install OFED We do not use IPoIB. Giovanni On 30/01/21 06:45, Walter Sklenka wrote: > Hi! > > Is it possible to mix OPAcards and Infininiband HCAs on the same server? > > In the faq > https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq. > html#rdma > > > They talk about RDMA : > > "RDMA is NOT supported on a node when both Mellanox HCAs and Intel > Omni-Path HFIs are ENABLED for RDMA." > > So do I understand right: When we do NOT enable the opa interface we > can still enable IB ? > > The reason I ask is, that we have a gpfs cluster of 6 NSD Servers > (wih access to storage) with opa interfaces which provide access to > remote cluster also via OPA. > > A new cluster with HDR interfaces will be implemented soon > > They shell have access to the same filesystems > > When we add HDR interfaces to NSD servers and enable rdma on this > network while disabling rdma on opa we would accept the worse > performance via opa . We hope that this provides still better perf > and less technical overhead than using routers > > Or am I totally wrong? > > Thank you very much and keep healthy! > > Best regards > > Walter > > Mit freundlichen Grüßen > */Walter Sklenka/* > */Technical Consultant/* > > EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 > Wien > Tel: +43 1 29 22 165-31 > Fax: +43 1 29 22 165-90 > E-Mail: skle...@edv-design.at <mailto:skle...@edv-design.at> > Internet: www.edv-design.at <http://www.edv-design.at/> > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- Giovanni Bracco phone +39 351 8804788 E-mail giovanni.bra...@enea.it WWW http://www.afs.enea.it/bracco _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss