Expected number of nodes for initial setup is 10-15 and of OSDs - 1,500-2,000.
Networking is planned to be 2 100GbE or 2 dual 50GbE in x16 slots (per OSD node). JBODs are to be connected with 3-4 x8 SAS3 HBAs (4 4x SAS3 ports each) Choice of hardware is done considering (non-trivial) per-server sw licensing costs - so small (12-24 HDD) nodes are certainly not optimal regardless of CPUs cost (which is estimated to be below 10% of the total cost in the setup I'm currently considering). EC (4+2 or 8+3 etc - TBD) - not 3x replication - is planned to be used for most of the storage space. Main applications are expected to be archiving and sequential access to large (multiGB) files/objects. Nick, which physical limitations you're referring to ? Thanks. On Sun, Dec 10, 2017 at 11:17 AM, Nick Fisk <[email protected]> wrote: > *From:* ceph-users [mailto:[email protected]] *On Behalf > Of *Igor Mendelev > *Sent:* 10 December 2017 15:39 > *To:* [email protected] > *Subject:* [ceph-users] what's the maximum number of OSDs per OSD server? > > > > Given that servers with 64 CPU cores (128 threads @ 2.7GHz) and up to 2TB > RAM - as well as 12TB HDDs - are easily available and somewhat reasonably > priced I wonder what's the maximum number of OSDs per OSD server (if using > 10TB or 12TB HDDs) and how much RAM does it really require if total storage > capacity for such OSD server is on the order of 1,000+ TB - is it still 1GB > RAM per TB of HDD or it could be less (during normal operations - and > extended with NVMe SSDs swap space for extra space during recovery)? > > > > Are there any known scalability limits in Ceph Luminous (12.2.2 with > BlueStore) and/or Linux that'll make such high capacity OSD server not > scale well (using sequential IO speed per HDD as a metric)? > > > > Thanks. > > > > How many total OSD’s will you have? If you are planning on having > thousands then dense nodes might make sense. Otherwise you are leaving > yourself open to having a few number of very large nodes, which will likely > shoot you in the foot further down the line. Also don’t forget, unless this > is purely for archiving, you will likely need to scale the networking up > per node, 2x10G won’t cut it when you have 10-20+ disks per node. > > > > With Bluestore, you are probably looking at around 2-3GB of RAM per OSD, > so say 4GB to be on the safe side. > > 7.2k HDD’s will likely only use a small proportion of a CPU core due to > their limited IO potential. A would imagine that even with 90 bay JBOD’s, > you will run into physical limitations before you hit CPU ones. > > > > Without knowing your exact requirements, I would suggest that larger > number of smaller nodes, might be a better idea. If you choose your > hardware right, you can often get the cost down to comparable levels by not > going with top of the range kit. Ie Xeon E3’s or D’s vs dual socket E5’s. >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
