What use-case (s) ? Are your pools R3, EC? Mix? I like to solve first for at least 9-10 nodes, but assuming that you’re using replicated size=3 pools 5 is okay.
Part of the answer is what these nodes look like. Are they dedicated to Ceph? How much RAM and CPU? Conventional wisdom is that when using NVMe SSDs to offload WAL+DB from HDDs you want one NVMe SSD to back at most 10x HDD OSDs. Do you have your 1TB NVMe SSDs dedicating 250GB to each of the 4 HDDs? Or do you have them sliced smaller? If you don’t have room on them for additional HDD OSDs that complicates the proposition. Sometimes people use PCIe to M.2 adapters to fit in additional NVMe drives, but take care to look into PCIe bifurcation etc. when selecting a card to accept more than one M.2 NVMe SSD. Are your NVMe drives enterprise-class? > On Mar 23, 2025, at 10:14 PM, Alan Murrell <[email protected]> wrote: > > Hello, > > We have a 5-node cluster that each have the following drives: > > * 4 x 16TB HDD > * 4 x 2TB NVME > * 1 x 1TB NVME (for the WAL/DB for the HDDs) > > The nodes don't have any more room to add more NVMEs, but they do have room > to add four more HDDs. I know adding more HDDs are able to make the cluster > faster due to the additional IOPs. > > So my question is this: > > Is it better to: > > * Add the additional drives/IOPs by adding an additional node > * Add the additional drives by adding the the HDDs to the existing nodes > > Or does it not really matter? I would prefer to add the drives to the > existing nodes (ultimately maxing them out) Please share what your nodes are like to inform suggestions. I’ve recently seen a cluster deployed with 8+2 EC on only 10 nodes and inadequate CPU. When things went pear-shaped it really, really wasn’t pretty. How many SAS/SATA drive bays do your nodes have for HDDs? Like most things in tech there are disagreements, but a rule of thumb is 2x vcores / threads per HDD OSD, 4-6 for NVMe OSDs. And extra for the OS, mons, mgrs, RGWs, etc. > , but just wondering if that affects performance as much as expanding by > adding additional nodes. > > Thanks! :-) > > Sent from my mobile device. Please excuse brevity and ttpos. > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
