Coming to the routing point, is there any reason why you need it ? I mean, this is because GPFS trying to connect between compute nodes or a reason outside GPFS scope ? If the reason is GPFS, imho best approach - without knowledge of the licensing you have - would be to use separate clusters: a storage cluster and two compute clusters.
Both compute clusters join using multicluster setup the storage cluster. There is no need both compute clusters see each other, they only need to see the storage cluster. One of the clusters using the 10G, the other cluster using the IPoIB interface. You need at least three quorum nodes in each compute cluster but if licensing is per drive on the DSS, it is covered. -- Jordi Caubet Serrabou IBM Software Defined Infrastructure (SDI) and Flash Technical Sales Specialist Technical Computing and HPC IT Specialist and Architect Ext. Phone: (+34) 679.79.17.84 (internal 55834) E-mail: [email protected] > On 5 Oct 2020, at 08:19, Olaf Weiser <[email protected]> wrote: > > > let me add a few comments from some very successful large installations in > Eruope > > # InterOP > Even though (as Luis pointed to) , there is no support statement to run > intermix DSS/ESS in general, it was ~, and is, and will be, ~ allowed for > short term purposes, such as e.g migration. > The reason to not support those DSS/ESS mixed configuration in general is > simply driven by the fact, that different release version of DSS/ESS > potentially (not in every release, but sometimes) comes with different > driver levels, (e.g. MOFED), OS, RDMA-settings, GPFS tuning, etc... > Those changes can have an impact/multiple impacts and therefore, we do not > support that in general. Of course -and this would be the advice for every > one - if you are faced the need to run a mixed configuration for e.g. a > migration and/or e.g. cause of you need to temporary provide space etc... > contact you IBM representative and settle to plan that accordingly.. > There will be (likely) some additional requirements/dependencies defined > like driver versions, OS, and/or Scale versions, but you'll get a chance to > run mixed configuration - temporary limited to your specific scenario. > > # Monitoring > No doubt, monitoring is essential and absolutely needed. - and/but - IBM > wants customers to be very sensitive, what kind of additional software > (=workload) gets installed on the ESS-IO servers. BTW, this rule applies as > well to any other important GPFS node with special roles (e.g. any other NSD > server etc) > But given the fact, that customer's usually manage and monitor their server > farms from a central point of control (any 3rd party software), it is common/ > best practice , that additionally monitor software(clients/endpoints) has to > run on GPFS nodes, so as on ESS nodes too. > > If that way of acceptance applies for DSS too, you may want to double check > with Lenovo ?! > > > #additionally GW functions > It would be a hot iron, to general allow routing on IO nodes. Similar to the > mixed support approach, the field variety for such a statement would be > hard(==impossible) to manage. As we all agree, additional network traffic can > (and in fact will) impact GPFS. > In your special case, the expected data rates seems to me more than ok and > acceptable to go with your suggested config (as long workloads remain on that > level / monitor it accordingly as you are already obviously doing) > Again,to be on the safe side.. contact your IBM representative and I'm sure > you 'll find a way.. > > > > kind regards.... > olaf > > > ----- Original message ----- > From: Jonathan Buzzard <[email protected]> > Sent by: [email protected] > To: [email protected] > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] Services on DSS/ESS nodes > Date: Sun, Oct 4, 2020 12:17 PM > > On 04/10/2020 10:29, Luis Bolinches wrote: > > Hi > > > > As stated on the same link you can do remote mounts from each other and > > be a supported setup. > > > > “ You can use the remote mount feature of IBM Spectrum Scale to share > > file system data across clusters.” > > > > You can, but imagine I have a DSS-G cluster, with 2PB of storage on it > which is quite modest in 2020. It is now end of life and for whatever > reason I decide I want to move to ESS instead. > > What any sane storage admin want to do at this stage is set the ESS, add > the ESS nodes to the existing cluster on the DSS-G then do a bit of > mmadddisk/mmdeldisk and sit back while the data is seemlessly moved from > the DSS-G to the ESS. Admittedly this might take a while :-) > > Then once all the data is moved a bit of mmdelnode and bingo the storage > has been migrated from DSS-G to ESS with zero downtime. > > As that is not allowed for what I presume are commercial reasons (you > could do it in reverse and presumable that is what IBM don't want) then > once you are down the rabbit hole of one type of storage the you are not > going to switch to a different one. > > You need to look at it from the perspective of the users. They frankly > could not give a monkeys what storage solution you are using. All they > care about is having usable storage and large amounts of downtime to > switch from one storage type to another is not really acceptable. > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > Salvo indicado de otro modo más arriba / Unless stated otherwise above: International Business Machines, S.A. Santa Hortensia, 26-28, 28002 Madrid Registro Mercantil de Madrid; Folio 1; Tomo 1525; Hoja M-28146 CIF A28-010791
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
