Hi,
On 8/7/25 19:14, Anthony D'Atri wrote:
Does anyone deployed CEPH in supermicro super storage nodes of 60/90 HDD + 4
NVME for WALL?
We do (supermicro 60 HDD nodes) in a 12 nodes cluster (20TB drives, 12
PiB raw) over 3 different server rooms / buildings. Our usecase is
CephFS only (replicated 3, metadata pool is backed with dedicated NVMe).
We didn't separate the WALL (they are on the spinners) but it's not a
bottleneck for our use case.
We're quite happy with this setup, even if it is not ideal. We had these
supermicro servers in stock already so we use them.
As Mark wrote, there are issues with dense toploaders.
* Dense toploaders are heavy, if you have a raised floor especially you may not
be able to put more than a couple in each rack safely.
* Same for power, a rack that can only be populated 25% full erodes any
perceived cost benefit
This is true, a fully populated 60 HDD supermicro node is ~100Kg. We
fill (for now) our racks with 4 nodes max, and some generic servers in
the remaining top half of the rack.
* If your cluster comprises a small number of these chassis, say <10, that's a
very, very large blast radius. When one of those nodes halts and catches fire, IF
you have sufficient spare capacity to heal, that process will be a thundering herd
that will degrade performance, and could easily take MONTH to complete. During
which time you are at increased risk of data unavailability or loss.
Supermicro super storage is interesting due to the possibility to swap
easily all the motherboard and active components without moving the
disks (tool-less removal of the back side of the server).
So we've got a spare server, and in case of a chassis failure, we're
able to restart (same OS/drives/OSDs...) with fresh hardware in a very
short amount of time. We've considered it's acceptable in our use-case.
In most cases you would be better off with a larger number of more
modestly-equipped servers, even if you stick with HDDs.
I agree (Clyso folks really know their stuff!), if we were to rebuild
this from scratch I would probably buy some 24 x 3.5" HDD per 2U nodes.
Still, money wise it may not be a dumb choice to use these supermicro
super storage nodes, depending of your needs and setup.
Cost is (was?) around €100k for 3 populated additional nodes. Not so bad
I think.
Anyway I'm interested in your feedback when you've made your choice.
Cheers,
Fabien
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
