Re: [gpfsug-discuss] WAS: alternative path; Now: RDMA

Jonathan: You posed a reasonable question, which was "when is RDMA worth the hassle?" I agree with part of your premises, which is that it only matters when the bottleneck isn't somewhere else. With a parallel file system, like Scale/GPFS, the absolute performance bottleneck is not the

Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration

On 09/12/2021 16:04, Douglas O'flaherty wrote: Though not directly about your design, our work with NVIDIA on GPUdirect Storage and SuperPOD has shown how sensitive RDMA (IB & RoCE) to both MOFED and Firmware version compatibility can be. I would suggest anyone debugging RDMA issues should

Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 119, Issue 7 - Adding a quorum node

On 09/12/2021 16:43, Ralf Eberhard wrote: Jonathan, my suspicion is that the GPFS daemon on fqdn-new is not reachable via port 1191. You can double check that by sending a lightweight CCR RPC to this daemon from another quorum node by attempting: mmccr echo -n fqdn-new;echo $? If this echo

Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration

Hello Douglas! Many thanks for your advice ! Well we are in a horrible situation regarding firmware and MOFED of old equipment Mellanox advised us to use a special version of subnetmanager 5.0-2.1.8.0 from MOFED I hope this helps Let´s see how we can proceed Best regards Walter From:

Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration

Hi Olaf!! Many thanks OK well we will do mmvdisk vs delete So #mmvdisk vs delete … -N ess01,ess02….. would be correct , or? Best regards walter From: gpfsug-discuss-boun...@spectrumscale.org On Behalf Of Olaf Weiser Sent: Donnerstag, 9. Dezember 2021 13:04 To:

Re: [gpfsug-discuss] Adding a quorum node

I frequently change quorum on the fly on both our 4.x and 5.0 clusters during upgrades/maintenance. You have sanity in the CCR to start with? (mmccr query, lsnodes, etc,etc) Anything useful in the logs or if you drop debug on it? ('export DEBUG=1'and then re-run command) Ed Wahl OSC

Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 119, Issue 7 - Adding a quorum node

rnate path between ESS Servers forDatamigrationMessage-ID:<of942131bb.73c8972f-on002587a6.0041d89c-002587a6.00425...@ibm.com>Content-Type: text/plain; charset="us-ascii"An HTML attachment was scrubbed...URL: <http://gpfsug.org/pipermail/gpfsug-discuss/a

Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration

Walter: Though not directly about your design, our work with NVIDIA on GPUdirect Storage and SuperPOD has shown how sensitive RDMA (IB & RoCE) to both MOFED and Firmware version compatibility can be. I would suggest anyone debugging RDMA issues should look at those closely. Doug by

[gpfsug-discuss] Adding a quorum node

I am looking to replace the quorum node in our cluster. The RAID card in the server we are currently using is a casualty of the RHEL8 SAS card purge :-( I have a "new" dual core server that is fully supported by RHEL8. After some toing and throwing with IBM they agreed a Pentium G6400 is

Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration

Hallo Walter, ;-) yes !AND! no ..   for sure , you can specifiy a subset of nodes to use RDMA and other nodes just communicating TCPIP But that's only half of the truth .   The other half is.. who and how , you are going to migrate/copy the data in case you 'll use mmrestripe you will have to

Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration

Hi Jan! That great to hear So we will try this Mit freundlichen Grüßen Walter Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210 Wien Tel: +43 1 29 22 165-31 Fax: +43 1 29 22 165-90 E-Mail: skle...@edv-design.at Internet:

Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration

I believe this should be a fully working solution. I see no problem enabling RDMA between a subset of nodes -- just disable verbsRdma on the nodes you want to use plain IP. -jf On Thu, Dec 9, 2021 at 11:04 AM Walter Sklenka wrote: > Dear spectrum scale users! > > May I ask you a design

[gpfsug-discuss] alternate path between ESS Servers for Datamigration

Dear spectrum scale users! May I ask you a design question? We have an IB environment which is very mixed at the moment ( connecX3 ... connect-X6 with FDR , even FDR10 and with arrive of ESS5000SC7 now also HDR100 and HDR switches. We still have some big troubles in this fabric when using RDMA