On 12/12/2021 02:19, Alec wrote:
I feel the need to respond here... I see many responses on this
User Group forum that are dismissive of the fringe / extreme use
cases and of the "what do you need that for '' mindset. The thing is
that Spectrum Scale is for the extreme, just take the word "Parallel"
in the old moniker that was already an extreme use case.
I wasn't been dismissive, I was asking what the benefits of using RDMA
where. There is very little information about it out there and not a lot
of comparative benchmarking on it either. Without the benefits being
clearly laid out I am unlikely to consider it and might be missing a trick.
IBM's literature on the topic is underwhelming to say the least.
[SNIP]
I have an AIX LPAR that traverses more than 300TB+ of data a day on a
Spectrum Scale file system, it is fully virtualized, and handles a
million files. If that performance level drops, regulatory reports
will be late, business decisions won't be current. However, the
systems of today and the future have to traverse this much data and
if they are slow then they can't keep up with real-time data feeds.
I have this nagging suspicion that modern all flash storage systems
could deliver that sort of performance without the overhead of a
parallel file system.
[SNIP]
Douglas's response is the right one, how much IO does the
application / environment need, it's nice to see Spectrum Scale have
the flexibility to deliver. I'm pretty confident that if I can't
deliver the required I/O performance on Spectrum Scale, nobody else
can on any other storage platform within reasonable limits.
I would note here that in our *shared HPC* environment I made a very
deliberate design decision to attach the compute nodes with 10Gbps
Ethernet for storage. Though I would probably pick 25Gbps if we where
procuring the system today.
There where many reasons behind that, but the main ones being that
historical file system performance showed that greater than 99% of the
time the file system never got above 20% of it's benchmarked speed.
Using 10Gbps Ethernet was not going to be a problem.
Secondly by limiting the connection to 10Gbps it stops one person
hogging the file system to the detriment of other users. We have seen
individual nodes peg their 10Gbps link from time to time, even several
nodes at once (jobs from the same user) and had they had access to a
100Gbps storage link that would have been curtains for everyone else's
file system usage.
At this juncture I would note that the GPFS admin traffic is handled by
on separate IP address space on a separate VLAN which we prioritize with
QOS on the switches. So even when a node floods it's 10Gbps link for
extended periods of time it doesn't get ejected from the cluster. The
need for a separate physical network for admin traffic is not necessary
in my experience.
That said you can do RDMA with Ethernet... Unfortunately the teaching
cluster and protocol nodes are on Intel X520's which I don't think do
RDMA. Everything is X710's or Mellanox Connect-X4 which definitely do do
RDMA. I could upgrade the protocol nodes but the teaching cluster would
be a problem.
JAB.
--
Jonathan A. Buzzard Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
