Re: [gpfsug-discuss] WAS: alternative path; Now: RDMA

[Douglas O'flaherty]

Jonathan:

You posed a reasonable question, which was "when is RDMA worth the 
hassle?"  I agree with part of your premises, which is that it only 
matters when the bottleneck isn't somewhere else. With a parallel file 
system, like Scale/GPFS, the absolute performance bottleneck is not the 
throughput of a single drive. In a majority of Scale/GPFS clusters the 
network data path is the performance limitation. If they deploy HDR or 
100/200/400Gbps Ethernet...  At that point, the buffer copy time inside 
the server matters. 

When the device is an accelerator, like a GPU, the benefit of RDMA (GDS) 
is easily demonstrated because it eliminates the bounce copy through the 
system memory. In our NVIDIA DGX A100 server testing testing we were able 
to get around 2x the per system throughput by using RDMA direct to GPU 
(GUP Direct Storage). (Tested on 2 DGX system with 4x HDR links per 
storage node.) 

However, your question remains. Synthetic benchmarks are good indicators 
of technical benefit, but do your users and applications need that extra 
performance? 

These are probably only a handful of codes in organizations that need 
this. However, they are high-value use cases. We have client applications 
that either read a lot of data semi-randomly and not-cached - think 
mini-Epics for scaling ML training. Or, demand lowest response time, like 
production inference on voice recognition and NLP. 

If anyone has use cases for GPU accelerated codes with truly demanding 
data needs, please reach out directly. We are looking for more use cases 
to characterize the benefit for a new paper. f you can provide some code 
examples, we can help test if RDMA direct to GPU (GPUdirect Storage) is a 
benefit. 

Thanks,

doug

Douglas O'Flaherty
dougla...@us.ibm.com






- Message from Jonathan Buzzard  on 
Fri, 10 Dec 2021 00:27:23 + -
To:
gpfsug-discuss@spectrumscale.org
Subject:
Re: [gpfsug-discuss] 
On 09/12/2021 16:04, Douglas O'flaherty wrote:
> 
> Though not directly about your design, our work with NVIDIA on GPUdirect 

> Storage and SuperPOD has shown how sensitive RDMA (IB & RoCE) to both 
> MOFED and Firmware version compatibility can be.
> 
> I would suggest anyone debugging RDMA issues should look at those 
closely.
> 
May I ask what are the alleged benefits of using RDMA in GPFS?

I can see there would be lower latency over a plain IP Ethernet or IPoIB 
solution but surely disk latency is going to swamp that?

I guess SSD drives might change that calculation but I have never seen 
proper benchmarks comparing the two, or even better yet all four 
connection options.

Just seems a lot of complexity and fragility for very little gain to me.


JAB.

-- 
Jonathan A. Buzzard Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG


 
 
- Original message -
From: "Jonathan Buzzard" 
Sent by: gpfsug-discuss-boun...@spectrumscale.org
To: gpfsug-discuss@spectrumscale.org
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] alternate path between ESS 
Servers for Datamigration
Date: Fri, Dec 10, 2021 10:27
 
On 09/12/2021 16:04, Douglas O'flaherty wrote:
>
> Though not directly about your design, our work with NVIDIA on GPUdirect
> Storage and SuperPOD has shown how sensitive RDMA (IB & RoCE) to both
> MOFED and Firmware version compatibility can be.
>
> I would suggest anyone debugging RDMA issues should look at those 
closely.
>
May I ask what are the alleged benefits of using RDMA in GPFS?

I can see there would be lower latency over a plain IP Ethernet or IPoIB
solution but surely disk latency is going to swamp that?

I guess SSD drives might change that calculation but I have never seen
proper benchmarks comparing the two, or even better yet all four
connection options.

Just seems a lot of complexity and fragility for very little gain to me.


JAB.

--
Jonathan A. Buzzard Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 




___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration

[Jonathan Buzzard]

On 09/12/2021 16:04, Douglas O'flaherty wrote:


Though not directly about your design, our work with NVIDIA on GPUdirect 
Storage and SuperPOD has shown how sensitive RDMA (IB & RoCE) to both 
MOFED and Firmware version compatibility can be.


I would suggest anyone debugging RDMA issues should look at those closely.


May I ask what are the alleged benefits of using RDMA in GPFS?

I can see there would be lower latency over a plain IP Ethernet or IPoIB 
solution but surely disk latency is going to swamp that?


I guess SSD drives might change that calculation but I have never seen 
proper benchmarks comparing the two, or even better yet all four 
connection options.


Just seems a lot of complexity and fragility for very little gain to me.


JAB.

--
Jonathan A. Buzzard Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 119, Issue 7 - Adding a quorum node

[Jonathan Buzzard]

On 09/12/2021 16:43, Ralf Eberhard wrote:

Jonathan,

my suspicion is that the GPFS daemon on fqdn-new is not reachable via 
port 1191.
You can double check that by sending a lightweight CCR RPC to this 
daemon from another quorum node by attempting:


mmccr echo -n fqdn-new;echo $?

If this echo returns with a non-zero exit code the network settings must 
be verified. And even the other direction must
work: Node fqdn-new must reach another quorum node, like (attempting on 
fqdn-new):


mmccr echo -n ;echo $?



Duh, that's my Homer Simpson moment for today.

I forgotten to move the relevant network interfaces on the new server to 
the trusted zone in the firewall. So of course my normal testing with 
ping and ssh was working just fine.



JAB.

--
Jonathan A. Buzzard Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration

[Walter Sklenka]

Hello Douglas!
Many thanks for your advice !
Well we are in a horrible situation  regarding firmware and MOFED of old 
equipment
Mellanox advised us to use a special version of subnetmanager  5.0-2.1.8.0 from 
MOFED
I hope this helps


Let´s see how we can proceed

Best regards
Walter


From: gpfsug-discuss-boun...@spectrumscale.org 
 On Behalf Of Douglas O'flaherty
Sent: Donnerstag, 9. Dezember 2021 17:04
To: gpfsug-discuss@spectrumscale.org
Subject: Re: [gpfsug-discuss] alternate path between ESS Servers for 
Datamigration

Walter:

Though not directly about your design, our work with NVIDIA on GPUdirect 
Storage and SuperPOD has shown how sensitive RDMA (IB & RoCE) to both MOFED and 
Firmware version compatibility can be.

I would suggest anyone debugging RDMA issues should look at those closely.

Doug


by carrier pigeon

On Dec 9, 2021, 5:04:36 AM, 
gpfsug-discuss-requ...@spectrumscale.org
 wrote:

From: 
gpfsug-discuss-requ...@spectrumscale.org
To: gpfsug-discuss@spectrumscale.org
Cc:
Date: Dec 9, 2021, 5:04:36 AM
Subject: [EXTERNAL] gpfsug-discuss Digest, Vol 119, Issue 5



Send gpfsug-discuss mailing list submissions to
gpfsug-discuss@spectrumscale.orgTo 
subscribe or unsubscribe via the World Wide Web, visit
http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message 
with subject or body 'help' to
gpfsug-discuss-request@spectrumscale.orgYou
 can reach the person managing the list at
gpfsug-discuss-owner@spectrumscale.orgWhen
 replying, please edit your Subject line so it is more specificthan "Re: 
Contents of gpfsug-discuss digest..."


Send gpfsug-discuss mailing list submissions to
gpfsug-discuss@spectrumscale.orgTo 
subscribe or unsubscribe via the World Wide Web, visit
http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message 
with subject or body 'help' to
gpfsug-discuss-request@spectrumscale.orgYou
 can reach the person managing the list at
gpfsug-discuss-owner@spectrumscale.orgWhen
 replying, please edit your Subject line so it is more specificthan "Re: 
Contents of gpfsug-discuss digest..."Today's Topics:   1. alternate path 
between ESS Servers forDatamigration  (Walter Sklenka)

Dear spectrum scale users!
May I ask you a design question?
We have an IB environment which is very mixed at the moment ( connecX3 … 
connect-X6 with FDR , even FDR10 and with arrive of ESS5000SC7 now also HDR100 
and HDR switches. We still have some big  troubles in this fabric when using 
RDMA , a case at Mellanox and IBM is open .
The environment has 3 old Building blocks 2xESSGL6 and 1x GL4 , from where we 
want to migrate the data to ess5000 , ( mmdelvdisk +qos)
Due to the current problems with RDMA we though eventually we could try a 
workaround :
If you are interested there is Maybe you can find the attachment ?
We build 2 separate fabrics , the ess-IO servers attached to both blue and 
green and all other cluster members and all remote clusters only to fabric blue
The daemon interfaces (IPoIP) are on fabric blue

It is the aim to setup rdma only on the ess-ioServers in the fabric green ,  in 
the blue we must use IPoIB (tcp)
Do you think datamigration would work between ess01,ess02,… to ess07,ess08 via 
RDMA ?
Or is it  principally not possible to make a rdma network only  for a subset of 
a cluster (though this subset would be reachable via other fabric) ?

Thank you very much for any input !
Best regards walter



Mit freundlichen Grüßen
Walter Sklenka
Technical Consultant

EDV-Design Informationstechnologie GmbH
Giefinggasse 6/1/2, A-1210 Wien
Tel: +43 1 29 22 165-31
Fax: +43 1 29 22 165-90
E-Mail: skle...@edv-design.at
Internet: www.edv-design.at


___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration

[Walter Sklenka]

Hi Olaf!!

Many thanks
OK well we will do mmvdisk vs delete
 So
#mmvdisk vs delete … -N ess01,ess02….. would be correct , or?

Best regards walter



From: gpfsug-discuss-boun...@spectrumscale.org 
 On Behalf Of Olaf Weiser
Sent: Donnerstag, 9. Dezember 2021 13:04
To: gpfsug-discuss@spectrumscale.org
Cc: gpfsug-discuss@spectrumscale.org
Subject: Re: [gpfsug-discuss] alternate path between ESS Servers for 
Datamigration

Hallo Walter,
;-)
yes !AND! no ..

for sure , you can specifiy a subset of nodes to use RDMA and other nodes just 
communicating TCPIP
But that's only half of the truth .

The other half is.. who and how , you are going to migrate/copy the data
in case you 'll use mmrestripe  you will have to make sure , that only 
nodes, connected(green) and configured for RDMA  doing the work
otherwise.. if will also work to migrate the data, but then data is send 
throught the Ethernet as well , (as long all those nodes are in the same 
cluster)


laff





- Ursprüngliche Nachricht -
Von: "Walter Sklenka" 
mailto:walter.skle...@edv-design.at>>
Gesendet von: 
gpfsug-discuss-boun...@spectrumscale.org
An: "'gpfsug-discuss@spectrumscale.org'" 
mailto:gpfsug-discuss@spectrumscale.org>>
CC:
Betreff: [EXTERNAL] [gpfsug-discuss] alternate path between ESS Servers for 
Datamigration
Datum: Do, 9. Dez 2021 11:04


Dear spectrum scale users!

May I ask you a design question?

We have an IB environment which is very mixed at the moment ( connecX3 … 
connect-X6 with FDR , even FDR10 and with arrive of ESS5000SC7 now also HDR100 
and HDR switches. We still have some big  troubles in this fabric when using 
RDMA , a case at Mellanox and IBM is open .

The environment has 3 old Building blocks 2xESSGL6 and 1x GL4 , from where we 
want to migrate the data to ess5000 , ( mmdelvdisk +qos)

Due to the current problems with RDMA we though eventually we could try a 
workaround :

If you are interested there is Maybe you can find the attachment ?

We build 2 separate fabrics , the ess-IO servers attached to both blue and 
green and all other cluster members and all remote clusters only to fabric blue

The daemon interfaces (IPoIP) are on fabric blue



It is the aim to setup rdma only on the ess-ioServers in the fabric green ,  in 
the blue we must use IPoIB (tcp)

Do you think datamigration would work between ess01,ess02,… to ess07,ess08 via 
RDMA ?

Or is it  principally not possible to make a rdma network only  for a subset of 
a cluster (though this subset would be reachable via other fabric) ?



Thank you very much for any input !

Best regards walter







Mit freundlichen Grüßen
Walter Sklenka
Technical Consultant



EDV-Design Informationstechnologie GmbH
Giefinggasse 6/1/2, A-1210 Wien
Tel: +43 1 29 22 165-31
Fax: +43 1 29 22 165-90
E-Mail: skle...@edv-design.at
Internet: www.edv-design.at



___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] Adding a quorum node

[Wahl, Edward]

I frequently change quorum on the fly on both our 4.x and 5.0 clusters during 
upgrades/maintenance.

You have sanity in the CCR to start with?  (mmccr query, lsnodes, etc,etc)
Anything useful in the logs or if you drop debug on it?  ('export DEBUG=1'and 
then re-run command)

Ed Wahl
OSC

-Original Message-
From: gpfsug-discuss-boun...@spectrumscale.org 
 On Behalf Of Jonathan Buzzard
Sent: Thursday, December 9, 2021 7:36 AM
To: 'gpfsug-discuss@spectrumscale.org' 
Subject: [gpfsug-discuss] Adding a quorum node


I am looking to replace the quorum node in our cluster. The RAID card in the 
server we are currently using is a casualty of the RHEL8 SAS card purge :-(

I have a "new" dual core server that is fully supported by RHEL8. After some 
toing and throwing with IBM they agreed a Pentium G6400 is 70PVU a core and two 
cores :-)  That said it is currently running RHEL7 because that's what the 
DSS-G nodes are running. The upgrade to RHEL8 is planned for next year.

Anyway I have added it into the GPFS cluster all well and good and GPFS is 
mounted just fine. However when I ran the command to make it a quorum node I 
got the following error (sanitized to remove actual DNS names and IP addresses

initialize (113, '', ('', 1191)) failed (err 79) server 
initialization failed (err 79)
mmchnode: Unexpected error from chnodes -n
1=:1191,2:1191,3=:1191,113=:1191 -f 1 -P
1191 .  Return code: 149
mmchnode: Unable to change the CCR quorum node configuration.
mmchnode: Command failed. Examine previous error messages to determine cause.

fqdn-new is the new node and fqdn1/2/3 are the existing quorum nodes. I want to 
remove fqdn3 in due course.

Anyone any idea what is going on? I thought you could change the quorum nodes 
on the fly?


JAB.

-- 
Jonathan A. Buzzard Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!KGKeukY!hO7wULtfr6n28eBJ0BB8sYyRMFo6Xl5_XDpsNZz3GiD_3nXlPf6nKHNR-X99$
 
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 119, Issue 7 - Adding a quorum node

[Ralf Eberhard]

Jonathan,my suspicion is that the GPFS daemon on fqdn-new is not reachable via port 1191.You can double check that by sending a lightweight CCR RPC to this daemon from another quorum node by attempting:mmccr echo -n fqdn-new;echo $?If this echo returns with a non-zero exit code the network settings must be verified. And even the other direction mustwork: Node fqdn-new must reach another quorum node, like (attempting on fqdn-new):mmccr echo -n ;echo $?Mit freundlichen Grüßen / Kind regardsRalf EberhardSpectrum Scale DeveloperIBM Systems - Dept. M929Mobile: +49 162 4159476E-Mail: ralf.eberh...@de.ibm.comIBM Deutschland Research & Development GmbHVorsitzender des Aufsichtsrats: Gregor PillenGeschäftsführung: Dirk WittkoppSitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294IBM Data Privacy Statement-gpfsug-discuss-boun...@spectrumscale.org wrote: -To: gpfsug-discuss@spectrumscale.orgFrom: gpfsug-discuss-requ...@spectrumscale.orgSent by: gpfsug-discuss-boun...@spectrumscale.orgDate: 12/09/2021 05:04PMSubject: [EXTERNAL] gpfsug-discuss Digest, Vol 119, Issue 7Send gpfsug-discuss mailing list submissions togpfsug-discuss@spectrumscale.orgTo subscribe or unsubscribe via the World Wide Web, visithttp://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via email, send a message with subject or body 'help' togpfsug-discuss-requ...@spectrumscale.orgYou can reach the person managing the list atgpfsug-discuss-ow...@spectrumscale.orgWhen replying, please edit your Subject line so it is more specificthan "Re: Contents of gpfsug-discuss digest..."Today's Topics:   1. Re: alternate path between ESS Servers forDatamigration      (Olaf Weiser)   2. Adding a quorum node (Jonathan Buzzard)   3. Re: alternate path between ESS Servers forDatamigration      (Douglas O'flaherty)--Message: 1Date: Thu, 9 Dec 2021 12:04:28 +From: "Olaf Weiser" <olaf.wei...@de.ibm.com>To: gpfsug-discuss@spectrumscale.orgCc: gpfsug-discuss@spectrumscale.orgSubject: Re: [gpfsug-discuss] alternate path between ESS Servers forDatamigrationMessage-ID:<of942131bb.73c8972f-on002587a6.0041d89c-002587a6.00425...@ibm.com>Content-Type: text/plain; charset="us-ascii"An HTML attachment was scrubbed...URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20211209/33805b31/attachment-0001.html >--Message: 2Date: Thu, 9 Dec 2021 12:36:08 +From: Jonathan Buzzard <jonathan.buzz...@strath.ac.uk>To: "'gpfsug-discuss@spectrumscale.org'"<gpfsug-discuss@spectrumscale.org>Subject: [gpfsug-discuss] Adding a quorum nodeMessage-ID: <73c81130-c120-d5f3-395f-4695e5690...@strath.ac.uk>Content-Type: text/plain; charset=UTF-8; format=flowedI am looking to replace the quorum node in our cluster. The RAID card in the server we are currently using is a casualty of the RHEL8 SAS card purge :-(I have a "new" dual core server that is fully supported by RHEL8. After some toing and throwing with IBM they agreed a Pentium G6400 is 70PVU a core and two cores :-)  That said it is currently running RHEL7 because that's what the DSS-G nodes are running. The upgrade to RHEL8 is planned for next year.Anyway I have added it into the GPFS cluster all well and good and GPFS is mounted just fine. However when I ran the command to make it a quorum node I got the following error (sanitized to remove actual DNS names and IP addressesinitialize (113, '', ('', 1191)) failed (err 79)server initialization failed (err 79)mmchnode: Unexpected error from chnodes -n 1=:1191,2:1191,3=:1191,113=:1191 -f 1 -P 1191 .  Return code: 149mmchnode: Unable to change the CCR quorum node configuration.mmchnode: Command failed. Examine previous error messages to determine cause.fqdn-new is the new node and fqdn1/2/3 are the existing quorum nodes. I want to remove fqdn3 in due course.Anyone any idea what is going on? I thought you could change the quorum nodes on the fly?JAB.-- Jonathan A. Buzzard                         Tel: +44141-5483420HPC System Administrator, ARCHIE-WeSt.University of Strathclyde, John Anderson Building, Glasgow. G4 0NG--Message: 3Date: Thu, 9 Dec 2021 16:04:28 +From: "Douglas O'flaherty" <dougla...@us.ibm.com>To: "gpfsug-discuss@spectrumscale.org"<gpfsug-discuss@spectrumscale.org>Subject: Re: [gpfsug-discuss] alternate path between ESS Servers forDatamigrationMessage-ID:<ofc5898fc6.442b7fc7-on002587a6.00584c7a-1639065868...@ibm.com>Content-Type: text/plain; charset="utf-8"Walt

Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration

[Douglas O'flaherty]

Walter:
 
 Though not directly about your design, our work with NVIDIA on GPUdirect 
Storage and SuperPOD has shown how sensitive RDMA (IB & RoCE) to both MOFED and 
Firmware version compatibility can be. 
 
 I would suggest anyone debugging RDMA issues should look at those closely. 
 
 Doug
 
 by carrier pigeon
 
 
   On Dec 9, 2021, 5:04:36 AM, gpfsug-discuss-requ...@spectrumscale.org wrote:
  
  From: gpfsug-discuss-requ...@spectrumscale.org
  To: gpfsug-discuss@spectrumscale.org
  Cc: 
  Date: Dec 9, 2021, 5:04:36 AM
  Subject: [EXTERNAL] gpfsug-discuss Digest, Vol 119, Issue 5
  
 Send gpfsug-discuss mailing list submissions to
gpfsug-discuss@spectrumscale.orgTo subscribe or unsubscribe via the World Wide 
Web, visit   http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via 
email, send a message with subject or body 'help' to  
gpfsug-discuss-request@spectrumscale.orgYou can reach the person managing the 
list at   gpfsug-discuss-owner@spectrumscale.orgWhen replying, please edit your 
Subject line so it is more specificthan "Re: Contents of gpfsug-discuss 
digest..."
   
   Send gpfsug-discuss mailing list submissions to  
gpfsug-discuss@spectrumscale.orgTo subscribe or unsubscribe via the World Wide 
Web, visit   http://gpfsug.org/mailman/listinfo/gpfsug-discuss or, via 
email, send a message with subject or body 'help' to  
gpfsug-discuss-request@spectrumscale.orgYou can reach the person managing the 
list at   gpfsug-discuss-owner@spectrumscale.orgWhen replying, please edit your 
Subject line so it is more specificthan "Re: Contents of gpfsug-discuss 
digest..."Today's Topics:   1. alternate path between ESS Servers for 
Datamigration  (Walter Sklenka)
 
Dear spectrum scale users! May I ask you a design question? We have an IB 
environment which is very mixed at the moment ( connecX3 … connect-X6 with FDR 
, even FDR10 and with arrive of ESS5000SC7 now also HDR100 and HDR switches. We 
still have some big  troubles in this fabric when using RDMA , a case at 
Mellanox and IBM is open .  The environment has 3 old Building blocks 2xESSGL6 
and 1x GL4 , from where we want to migrate the data to ess5000 , ( mmdelvdisk 
+qos)  Due to the current problems with RDMA we though eventually we could try 
a workaround : If you are interested there is Maybe you can find the attachment 
?  We build 2 separate fabrics , the ess-IO servers attached to both blue and 
green and all other cluster members and all remote clusters only to fabric blue 
 The daemon interfaces (IPoIP) are on fabric blue   It is the aim to setup rdma 
only on the ess-ioServers in the fabric green ,  in the blue we must use IPoIB 
(tcp)  Do you think datamigration would work between ess01,ess02,… to 
ess07,ess08 via RDMA ? Or is it  principally not possible to make a rdma 
network only  for a subset of a cluster (though this subset would be reachable 
via other fabric) ?   Thank you very much for any input ! Best regards walter   
 Mit freundlichen Grüßen
 Walter Sklenka
 Technical ConsultantEDV-Design Informationstechnologie GmbH
 Giefinggasse 6/1/2, A-1210 Wien
 Tel: +43 1 29 22 165-31
 Fax: +43 1 29 22 165-90
 E-Mail: skle...@edv-design.at
 Internet: www.edv-design.at   

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] Adding a quorum node

[Jonathan Buzzard]

I am looking to replace the quorum node in our cluster. The RAID card in 
the server we are currently using is a casualty of the RHEL8 SAS card 
purge :-(


I have a "new" dual core server that is fully supported by RHEL8. After 
some toing and throwing with IBM they agreed a Pentium G6400 is 70PVU a 
core and two cores :-)  That said it is currently running RHEL7 because 
that's what the DSS-G nodes are running. The upgrade to RHEL8 is planned 
for next year.


Anyway I have added it into the GPFS cluster all well and good and GPFS 
is mounted just fine. However when I ran the command to make it a quorum 
node I got the following error (sanitized to remove actual DNS names and 
IP addresses


initialize (113, '', ('', 1191)) failed (err 79)
server initialization failed (err 79)
mmchnode: Unexpected error from chnodes -n 
1=:1191,2:1191,3=:1191,113=:1191 -f 1 -P 
1191 .  Return code: 149

mmchnode: Unable to change the CCR quorum node configuration.
mmchnode: Command failed. Examine previous error messages to determine 
cause.


fqdn-new is the new node and fqdn1/2/3 are the existing quorum nodes. I 
want to remove fqdn3 in due course.


Anyone any idea what is going on? I thought you could change the quorum 
nodes on the fly?



JAB.

--
Jonathan A. Buzzard Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration

[Olaf Weiser]

Hallo Walter,
;-)
yes !AND! no ..
 
for sure , you can specifiy a subset of nodes to use RDMA and other nodes just communicating TCPIP
But that's only half of the truth .
 
The other half is.. who and how , you are going to migrate/copy the data
in case you 'll use mmrestripe  you will have to make sure , that only nodes, connected(green) and configured for RDMA  doing the work
otherwise.. if will also work to migrate the data, but then data is send throught the Ethernet as well , (as long all those nodes are in the same cluster)
 
 
laff
 
 
 
 
 
- Ursprüngliche Nachricht -Von: "Walter Sklenka" Gesendet von: gpfsug-discuss-boun...@spectrumscale.orgAn: "'gpfsug-discuss@spectrumscale.org'" CC:Betreff: [EXTERNAL] [gpfsug-discuss] alternate path between ESS Servers for DatamigrationDatum: Do, 9. Dez 2021 11:04  
Dear spectrum scale users!
May I ask you a design question?
We have an IB environment which is very mixed at the moment ( connecX3 … connect-X6 with FDR , even FDR10 and with arrive of ESS5000SC7 now also HDR100 and HDR switches. We still have some big  troubles in this fabric when using RDMA , a case at Mellanox and IBM is open . 
The environment has 3 old Building blocks 2xESSGL6 and 1x GL4 , from where we want to migrate the data to ess5000 , ( mmdelvdisk +qos) 
Due to the current problems with RDMA we though eventually we could try a workaround :
If you are interested there is Maybe you can find the attachment ? 
We build 2 separate fabrics , the ess-IO servers attached to both blue and green and all other cluster members and all remote clusters only to fabric blue 
The daemon interfaces (IPoIP) are on fabric blue
 
It is the aim to setup rdma only on the ess-ioServers in the fabric green ,  in the blue we must use IPoIB (tcp) 
Do you think datamigration would work between ess01,ess02,… to ess07,ess08 via RDMA ?
Or is it  principally not possible to make a rdma network only  for a subset of a cluster (though this subset would be reachable via other fabric) ?
 
Thank you very much for any input !
Best regards walter 
 
 
 
Mit freundlichen GrüßenWalter SklenkaTechnical Consultant 
 
EDV-Design Informationstechnologie GmbHGiefinggasse 6/1/2, A-1210 WienTel: +43 1 29 22 165-31Fax: +43 1 29 22 165-90E-Mail: skle...@edv-design.atInternet: www.edv-design.at
 
 
___gpfsug-discuss mailing listgpfsug-discuss at spectrumscale.orghttp://gpfsug.org/mailman/listinfo/gpfsug-discuss 
 

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration

[Walter Sklenka]

Hi Jan!
That great to hear
So we will try this

Mit freundlichen Grüßen
Walter Sklenka
Technical Consultant

EDV-Design Informationstechnologie GmbH
Giefinggasse 6/1/2, A-1210 Wien
Tel: +43 1 29 22 165-31
Fax: +43 1 29 22 165-90
E-Mail: skle...@edv-design.at
Internet: www.edv-design.at

Von: Jan-Frode Myklebust 
Gesendet: Thursday, December 9, 2021 11:25 AM
An: Walter Sklenka 
Cc: gpfsug-discuss@spectrumscale.org
Betreff: Re: [gpfsug-discuss] alternate path between ESS Servers for 
Datamigration

I believe this should be a fully working solution. I see no problem enabling 
RDMA between a subset of nodes -- just disable verbsRdma on the nodes you want 
to use plain IP.


  -jf

On Thu, Dec 9, 2021 at 11:04 AM Walter Sklenka 
mailto:walter.skle...@edv-design.at>> wrote:
Dear spectrum scale users!
May I ask you a design question?
We have an IB environment which is very mixed at the moment ( connecX3 … 
connect-X6 with FDR , even FDR10 and with arrive of ESS5000SC7 now also HDR100 
and HDR switches. We still have some big  troubles in this fabric when using 
RDMA , a case at Mellanox and IBM is open .
The environment has 3 old Building blocks 2xESSGL6 and 1x GL4 , from where we 
want to migrate the data to ess5000 , ( mmdelvdisk +qos)
Due to the current problems with RDMA we though eventually we could try a 
workaround :
If you are interested there is Maybe you can find the attachment ?
We build 2 separate fabrics , the ess-IO servers attached to both blue and 
green and all other cluster members and all remote clusters only to fabric blue
The daemon interfaces (IPoIP) are on fabric blue

It is the aim to setup rdma only on the ess-ioServers in the fabric green ,  in 
the blue we must use IPoIB (tcp)
Do you think datamigration would work between ess01,ess02,… to ess07,ess08 via 
RDMA ?
Or is it  principally not possible to make a rdma network only  for a subset of 
a cluster (though this subset would be reachable via other fabric) ?

Thank you very much for any input !
Best regards walter



Mit freundlichen Grüßen
Walter Sklenka
Technical Consultant

EDV-Design Informationstechnologie GmbH
Giefinggasse 6/1/2, A-1210 Wien
Tel: +43 1 29 22 165-31
Fax: +43 1 29 22 165-90
E-Mail: skle...@edv-design.at
Internet: www.edv-design.at

___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration

[Jan-Frode Myklebust]

I believe this should be a fully working solution. I see no problem
enabling RDMA between a subset of nodes -- just disable verbsRdma on the
nodes you want to use plain IP.


  -jf

On Thu, Dec 9, 2021 at 11:04 AM Walter Sklenka 
wrote:

> Dear spectrum scale users!
>
> May I ask you a design question?
>
> We have an IB environment which is very mixed at the moment ( connecX3 …
> connect-X6 with FDR , even FDR10 and with arrive of ESS5000SC7 now also
> HDR100 and HDR switches. We still have some big  troubles in this fabric
> when using RDMA , a case at Mellanox and IBM is open .
>
> The environment has 3 old Building blocks 2xESSGL6 and 1x GL4 , from where
> we want to migrate the data to ess5000 , ( mmdelvdisk +qos)
>
> Due to the current problems with RDMA we though eventually we could try a
> workaround :
>
> If you are interested there is Maybe you can find the attachment ?
>
> We build 2 separate fabrics , the ess-IO servers attached to both blue and
> green and all other cluster members and all remote clusters only to fabric
> blue
>
> The daemon interfaces (IPoIP) are on fabric blue
>
>
>
> It is the aim to setup rdma only on the ess-ioServers in the fabric green
> ,  in the blue we must use IPoIB (tcp)
>
> Do you think datamigration would work between ess01,ess02,… to ess07,ess08
> via RDMA ?
>
> Or is it  principally not possible to make a rdma network only  for a
> subset of a cluster (though this subset would be reachable via other
> fabric) ?
>
>
>
> Thank you very much for any input !
>
> Best regards walter
>
>
>
>
>
>
>
> Mit freundlichen Grüßen
> *Walter Sklenka*
> *Technical Consultant*
>
>
>
> EDV-Design Informationstechnologie GmbH
> Giefinggasse 6/1/2, A-1210 Wien
> Tel: +43 1 29 22 165-31
> Fax: +43 1 29 22 165-90
> E-Mail: skle...@edv-design.at
> Internet: www.edv-design.at
>
>
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

[gpfsug-discuss] alternate path between ESS Servers for Datamigration

[Walter Sklenka]

Dear spectrum scale users!
May I ask you a design question?
We have an IB environment which is very mixed at the moment ( connecX3 ... 
connect-X6 with FDR , even FDR10 and with arrive of ESS5000SC7 now also HDR100 
and HDR switches. We still have some big  troubles in this fabric when using 
RDMA , a case at Mellanox and IBM is open .
The environment has 3 old Building blocks 2xESSGL6 and 1x GL4 , from where we 
want to migrate the data to ess5000 , ( mmdelvdisk +qos)
Due to the current problems with RDMA we though eventually we could try a 
workaround :
If you are interested there is Maybe you can find the attachment ?
We build 2 separate fabrics , the ess-IO servers attached to both blue and 
green and all other cluster members and all remote clusters only to fabric blue
The daemon interfaces (IPoIP) are on fabric blue

It is the aim to setup rdma only on the ess-ioServers in the fabric green ,  in 
the blue we must use IPoIB (tcp)
Do you think datamigration would work between ess01,ess02,... to ess07,ess08 
via RDMA ?
Or is it  principally not possible to make a rdma network only  for a subset of 
a cluster (though this subset would be reachable via other fabric) ?

Thank you very much for any input !
Best regards walter



Mit freundlichen Grüßen
Walter Sklenka
Technical Consultant

EDV-Design Informationstechnologie GmbH
Giefinggasse 6/1/2, A-1210 Wien
Tel: +43 1 29 22 165-31
Fax: +43 1 29 22 165-90
E-Mail: skle...@edv-design.at
Internet: www.edv-design.at



Visio-eodc-2-fabs.pdf
Description: Visio-eodc-2-fabs.pdf
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss