I was only interjecting on the comment "So that is 5 . Which is real easy to obtain" and commenting on what the sustained writes into a cluster of 2,000 OSDs would require to actually sustain that 5 MBps on each SSD journal.
My calculation was off because I forgot replica size, but my corrected math is this... 5 MBps per journal device 8 OSDs per journal (overestimated number as most do 4) 2,000 OSDs based on what you said "Which is real easy to obtain, even with hardware 0f 2000." 3 replicas 2,000 OSDs / 8 OSDs per journal = 250 journal SSDs 250 SSDs * 5 MBps = 1,250 MBps / 3 replicas = 416.67 MBps required sustained cluster write speed to cause each SSD to average 5 MBps on each journal device. Swap out any variable you want to match your environment. For example, if you only have 4 OSDs per journal device, that number would be double for a cluster this size to require a cluster write speed of 833.33 MBps to average 5 MBps on each journal. Also if you have less than 2,000 OSDs, then everything shrinks fast. On Tue, May 2, 2017 at 5:39 PM Willem Jan Withagen <[email protected]> wrote: > On 02-05-17 19:54, David Turner wrote: > > Are you guys talking about 5Mbytes/sec to each journal device? Even if > > you had 8 OSDs per journal and had 2000 osds... you would need a > > sustained 1.25 Gbytes/sec to average 5Mbytes/sec per journal device. > > I'm not sure I'm following this... > But I'm rather curious. > Are you saying that the required journal bandwidth versus OSD write > bandwidth has an approx 1:200 ratio?? > > Note that I took it the other way. > Given the Intel specs > - What sustained bandwidth is allowed to have the device last its > lifetime. > - How much more usage would a 3710 give in regards to a 3520 SSD per > dollar spent. > > --WjW > > > On Tue, May 2, 2017 at 1:47 PM Willem Jan Withagen <[email protected] > > <mailto:[email protected]>> wrote: > > > > On 02-05-17 19:16, Дробышевский, Владимир wrote: > > > Willem, > > > > > > please note that you use 1.6TB Intel S3520 endurance rating in > your > > > calculations but then compare prices with 480GB model, which has > only > > > 945TBW or 1.1DWPD ( > > > > > > https://ark.intel.com/products/93026/Intel-SSD-DC-S3520-Series-480GB-2_5in-SATA-6Gbs-3D1-MLC > > > ). It also worth to notice that S3710 has tremendously higher write > > > speed\IOPS and especially SYNC writes. Haven't seen S3520 real sync > > > write tests yet but don't think they differ much from S3510 ones. > > > > Arrgh, you are right. I guess I had too many pages open, and copied > the > > wrong one. > > > > But the good news is that the stats were already in favour of the > 3710 > > so this only increases that conclusion. > > > > The bad news is that the sustained write speed goes down with a > > factor 4. > > So that is 5Mbyte/sec. Which is real easy to obtain, even with > hardware > > 0f 2000. > > > > --WjW > > > > > > > Best regards, > > > Vladimir > > > > > > 2017-05-02 21:05 GMT+05:00 Willem Jan Withagen <[email protected] > > <mailto:[email protected]> > > > <mailto:[email protected] <mailto:[email protected]>>>: > > > > > > On 27-4-2017 20:46, Alexandre DERUMIER wrote: > > > > Hi, > > > > > > > >>> What I'm trying to get from the list is /why/ the > > "enterprise" drives > > > >>> are important. Performance? Reliability? Something else? > > > > > > > > performance, for sure (for SYNC write, > > > https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ > > > > > < > https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ > >) > > > > > > > > Reliabity : yes, enteprise drive have supercapacitor in case > > of powerfailure, and endurance (1 DWPD for 3520, 3 DWPD for 3610) > > > > > > > > > > > >>> Also, 4 x Intel DC S3520 costs as much as 1 x Intel DC > > S3610. Obviously > > > >>> the single drive leaves more bays free for OSD disks, but > > is there any > > > >>> other reason a single S3610 is preferable to 4 S3520s? > > Wouldn't 4xS3520s > > > >>> mean: > > > > > > > > where do you see this price difference ? > > > > > > > > for me , S3520 are around 25-30% cheaper than S3610 > > > > > > I just checked for the DCS3520 on > > > > > > https://ark.intel.com/nl/products/93005/Intel-SSD-DC-S3520-Series-1_6TB-2_5in-SATA-6Gbs-3D1-MLC > > > > > < > https://ark.intel.com/nl/products/93005/Intel-SSD-DC-S3520-Series-1_6TB-2_5in-SATA-6Gbs-3D1-MLC > > > > > > > > And is has a TBW of 2925 (Terrabytes Write over life time) = > > 2,9 PB > > > the warranty is 5 years. > > > > > > Now if I do the math: > > > 2925 * 104 /5 /365 /24 /60 = 1,14 Gbyte/min to be written. > > > which is approx 20Mbyte /sec > > > or approx 10Gbit/min = 0,15 Gbit/sec > > > > > > And that is only 20% of the capacity of that SATA link. > > > Also writing 20Mbyte/sec sustained is not really that hard for > > modern > > > systems. > > > > > > Now a 400Gb 3710 takes 8.3 PB, which is ruffly 3 times as much. > > > so it will last 3 times longer. > > > > > > Checking Amazone, I get > > > $520 for the DC S3710-400G > > > $300 for the DC S3520-480G > > > > > > So that is less than a factor of 2 for using the S3710's and a > > 3 times > > > longer lifetime. To be exact (8.3/520) / (2,9/300) = 1.65 more > > bang for > > > your buck. > > > > > > But still do not expect your SSDs to last very long if the > > write rate is > > > much over that 20Mbyte/sec > > > > > > --WjW > > > > > > > > > > > > _______________________________________________ > > > ceph-users mailing list > > > [email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected] > >> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> > > > > > > > > > > > > > > > -- > > > > > > С уважением, > > > Дробышевский Владимир > > > Компания "АйТи Город" > > > +7 343 2222192 <+7%20343%20222-21-92> <tel:+7%20343%20222-21-92> > > > > > > ИТ-консалтинг > > > Поставка проектов "под ключ" > > > Аутсорсинг ИТ-услуг > > > Аутсорсинг ИТ-инфраструктуры > > > > _______________________________________________ > > ceph-users mailing list > > [email protected] <mailto:[email protected]> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
