IIRC, the filesystem descriptor disk is only in the system pool… so as long as the system pool only has 3 FGs and they correspond to your 3 sites, then the filesystem survivability characteristics are straightforward.
I think that you technically could use two different FGs for the second pool and GPFS will still work as expected, but that just seems confusing to the humans. We started off with one multisite stretch cluster like you describe and 10 years later we have around 100 stretch clusters. Choosing a standard mapping between FG numbers and your sites can be a good way to reduce cognitive load on your team. -Paul From: gpfsug-discuss <[email protected]> On Behalf Of Luke Sudbery Sent: Tuesday, March 18, 2025 16:18 To: gpfsug main discussion list <[email protected]> Subject: Re: [gpfsug-discuss] Replicated cluster - failure groups This message was sent by an external party. But any pros/cons of 5 (2 failure groups per site + tiebreaker ) vs 3? If we had 1 failure group per site, would we need to bring up all NSDs on that site (4x DSS – 8 actual servers) to guarantee bringing up the NSD with the desc replica disk? Cheers, Luke -- Luke Sudbery Principal Engineer (HPC and Storage). Architecture, Infrastructure and Systems Advanced Research Computing, IT Services Room 132, Computer Centre G5, Elms Road Please note I don’t work on Monday. From: gpfsug-discuss <[email protected]<mailto:[email protected]>> On Behalf Of scale Sent: 18 March 2025 17:21 To: gpfsug main discussion list <[email protected]<mailto:[email protected]>> Subject: Re: [gpfsug-discuss] Replicated cluster - failure groups CAUTION: This email originated from outside the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. There is no advantage to having more failure group than maximum number of replica supported by a file system plus 1 for tie breaker disks. In a multiple site setup, you will want 1 failure group per site in order to ensure 1 replica is placed at each site as GPFS will place replica using round-robin amount the failure groups. From: gpfsug-discuss <[email protected]<mailto:[email protected]>> on behalf of Luke Sudbery <[email protected]<mailto:[email protected]>> Date: Tuesday, March 18, 2025 at 9:28 AM To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: [EXTERNAL] [gpfsug-discuss] Replicated cluster - failure groups We are planning a replicated cluster. Due to a combination of purchasing cycles, floor loading and VAT-exemption status for half the equipment/data, this will be built over time using a total 8 Lenovo DSS building blocks. 2 main pools, in 2 We are planning a replicated cluster. Due to a combination of purchasing cycles, floor loading and VAT-exemption status for half the equipment/data, this will be built over time using a total 8 Lenovo DSS building blocks. 2 main pools, in 2 data centres, with 2 DSSG per pool, and a quorum/manager node with a local tie breaker disk in a 3rd physical location. My main question is about failure groups - so far, with 2 DSS and 1 tiebreaker, we would have had 1 failure group per DSS and 1 for the tie breaker disk, giving us a total of 3. But if we did that now we would have 9 failure groups in 1 filesystem, which is more than the maximum number of replicas of the file system descriptor and not desirable, as I understand it. So we could have either: * 1 FG per physical site, and assign all 4 DSS per site to 1 FG, and a 3rd to the tiebreaker * 1 FG per pool per site, with 2 DSS in each FG. This makes sense as the pairs of DSSG will both always need to be up for all the data in the pool to be accessible. The second option would give us 5 failure groups, but what would be the advantage and disadvantages of more failure groups? Many thanks, Luke -- Luke Sudbery Principal Engineer (HPC and Storage). Architecture, Infrastructure and Systems Advanced Research Computing, IT Services Room 132, Computer Centre G5, Elms Road Please note I don’t work on Monday.
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
