Re: [ceph-users] Consumer-grade SSD in Ceph
We didn’t find a measurable difference doing this on 5100s, ymmv. It depends on the controller... With chipset SATA and LSI 9200 HBA the difference is huge. I have some evidence here: https://yourcmc.ru/wiki/Ceph_performance#Server_SSDs With some controllers it may be not the case. -- With best regards, Vitaliy Filippov ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
>> SATA: Micron 5100-5200-5300, Seagate Nytro 1351/1551 (don't forget to >> disable their cache with hdparm -W 0) We didn’t find a measurable difference doing this on 5100s, ymmv. Depending on your use-case, CRUSH rules (EC vs R), etc. sub-DWPD models may be fine for OSDs, but I suggest higher durability for mon DBs. I thought this list was dead and we were using the ceph.io list now? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
Also, just for more diversity, Samsung has the 883 DCT and the 860 DCT models as well. Both less than 1 DWPD, but they are enterprise rated. Reed > On Jan 3, 2020, at 2:10 AM, Eneko Lacunza wrote: > > I'm sure you know also the following, but just in case: > - Intel SATA D3-S4610 (I think they're out of stock right now) > - Intel SATA D3-S4510 (I see stock of these right now) > > El 27/12/19 a las 17:56, vita...@yourcmc.ru escribió: >> SATA: Micron 5100-5200-5300, Seagate Nytro 1351/1551 (don't forget to >> disable their cache with hdparm -W 0) >> >> NVMe: Intel P4500, Micron 9300 >> >>> Thanks for all the replies. In summary; consumer grade SSD is a no go. >>> >>> What is an alternative to SM863a? Since it is quite hard to get these >>> due non non-stock. > > > -- > Zuzendari Teknikoa / Director Técnico > Binovo IT Human Project, S.L. > Telf. 943569206 > Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa) > www.binovo.es > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com smime.p7s Description: S/MIME cryptographic signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
I'm sure you know also the following, but just in case: - Intel SATA D3-S4610 (I think they're out of stock right now) - Intel SATA D3-S4510 (I see stock of these right now) El 27/12/19 a las 17:56, vita...@yourcmc.ru escribió: SATA: Micron 5100-5200-5300, Seagate Nytro 1351/1551 (don't forget to disable their cache with hdparm -W 0) NVMe: Intel P4500, Micron 9300 Thanks for all the replies. In summary; consumer grade SSD is a no go. What is an alternative to SM863a? Since it is quite hard to get these due non non-stock. -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
Quoting Sinan Polat (si...@turka.nl): > Thanks for all the replies. In summary; consumer grade SSD is a no go. > > What is an alternative to SM863a? Since it is quite hard to get these due non > non-stock. PM863a ... lower endurance ... but still "enterprise" ... but as your not concerned about lifetime this is just fine. We use quite a lot of them and even after ~ 2 years the most used SSD is at 4.4% write capacity. Gr. Stefan -- | BIT BV https://www.bit.nl/Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / i...@bit.nl ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
SATA: Micron 5100-5200-5300, Seagate Nytro 1351/1551 (don't forget to disable their cache with hdparm -W 0) NVMe: Intel P4500, Micron 9300 Thanks for all the replies. In summary; consumer grade SSD is a no go. What is an alternative to SM863a? Since it is quite hard to get these due non non-stock. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
Micron 9300 Obtener Outlook para Android<https://aka.ms/ghei36> From: ceph-users on behalf of Sinan Polat Sent: Friday, December 27, 2019 12:35:44 PM To: Eneko Lacunza Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Consumer-grade SSD in Ceph Thanks for all the replies. In summary; consumer grade SSD is a no go. What is an alternative to SM863a? Since it is quite hard to get these due non non-stock. Thanks! Sinan > Op 23 dec. 2019 om 08:50 heeft Eneko Lacunza het > volgende geschreven: > > Hi Sinan, > > Just to reiterate: don't do this. Consumer SSDs will destroy your enterprise > SSD's performance. > > Our office cluster is made of consumer-grade servers: cheap gaming > motherboards, memory, ryzen processors, desktop HDDs. But SSD drives are > Enterprise, we had awful experiences with consumer SSDs (some perform worse > that HDDs with Ceph). > > Cheers > Eneko > >> El 19/12/19 a las 20:20, Sinan Polat escribió: >> Hi all, >> >> Thanks for the replies. I am not worried about their lifetime. We will be >> adding only 1 SSD disk per physical server. All SSD’s are enterprise drives. >> If the added consumer grade disk will fail, no problem. >> >> I am more curious regarding their I/O performance. I do want to have 50% >> drop in performance. >> >> So anyone any experience with 860 EVO or Crucial MX500 in a Ceph setup? >> >> Thanks! >> >>> Op 19 dec. 2019 om 19:18 heeft Mark Nelson het >>> volgende geschreven: >>> >>> The way I try to look at this is: >>> >>> >>> 1) How much more do the enterprise grade drives cost? >>> >>> 2) What are the benefits? (Faster performance, longer life, etc) >>> >>> 3) How much does it cost to deal with downtime, diagnose issues, and >>> replace malfunctioning hardware? >>> >>> >>> My personal take is that enterprise drives are usually worth it. There may >>> be consumer grade drives that may be worth considering in very specific >>> scenarios if they still have power loss protection and high write >>> durability. Even when I was in academia years ago with very limited >>> budgets, we got burned with consumer grade SSDs to the point where we had >>> to replace them all. You have to be very careful and know exactly what you >>> are buying. >>> >>> >>> Mark >>> >>> >>>> On 12/19/19 12:04 PM, jes...@krogh.cc wrote: >>>> I dont think “usually” is good enough in a production setup. >>>> >>>> >>>> >>>> Sent from myMail for iOS >>>> >>>> >>>> Thursday, 19 December 2019, 12.09 +0100 from Виталий Филиппов >>>> : >>>> >>>>Usually it doesn't, it only harms performance and probably SSD >>>>lifetime >>>>too >>>> >>>>> I would not be running ceph on ssds without powerloss protection. I >>>>> delivers a potential data loss scenario >>>> >>>> >>>> ___ >>>> ceph-users mailing list >>>> ceph-users@lists.ceph.com >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > -- > Zuzendari Teknikoa / Director Técnico > Binovo IT Human Project, S.L. > Telf. 943569206 > Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa) > www.binovo.es<http://www.binovo.es> > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
Thanks for all the replies. In summary; consumer grade SSD is a no go. What is an alternative to SM863a? Since it is quite hard to get these due non non-stock. Thanks! Sinan > Op 23 dec. 2019 om 08:50 heeft Eneko Lacunza het > volgende geschreven: > > Hi Sinan, > > Just to reiterate: don't do this. Consumer SSDs will destroy your enterprise > SSD's performance. > > Our office cluster is made of consumer-grade servers: cheap gaming > motherboards, memory, ryzen processors, desktop HDDs. But SSD drives are > Enterprise, we had awful experiences with consumer SSDs (some perform worse > that HDDs with Ceph). > > Cheers > Eneko > >> El 19/12/19 a las 20:20, Sinan Polat escribió: >> Hi all, >> >> Thanks for the replies. I am not worried about their lifetime. We will be >> adding only 1 SSD disk per physical server. All SSD’s are enterprise drives. >> If the added consumer grade disk will fail, no problem. >> >> I am more curious regarding their I/O performance. I do want to have 50% >> drop in performance. >> >> So anyone any experience with 860 EVO or Crucial MX500 in a Ceph setup? >> >> Thanks! >> >>> Op 19 dec. 2019 om 19:18 heeft Mark Nelson het >>> volgende geschreven: >>> >>> The way I try to look at this is: >>> >>> >>> 1) How much more do the enterprise grade drives cost? >>> >>> 2) What are the benefits? (Faster performance, longer life, etc) >>> >>> 3) How much does it cost to deal with downtime, diagnose issues, and >>> replace malfunctioning hardware? >>> >>> >>> My personal take is that enterprise drives are usually worth it. There may >>> be consumer grade drives that may be worth considering in very specific >>> scenarios if they still have power loss protection and high write >>> durability. Even when I was in academia years ago with very limited >>> budgets, we got burned with consumer grade SSDs to the point where we had >>> to replace them all. You have to be very careful and know exactly what you >>> are buying. >>> >>> >>> Mark >>> >>> On 12/19/19 12:04 PM, jes...@krogh.cc wrote: I dont think “usually” is good enough in a production setup. Sent from myMail for iOS Thursday, 19 December 2019, 12.09 +0100 from Виталий Филиппов : Usually it doesn't, it only harms performance and probably SSD lifetime too > I would not be running ceph on ssds without powerloss protection. I > delivers a potential data loss scenario ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > -- > Zuzendari Teknikoa / Director Técnico > Binovo IT Human Project, S.L. > Telf. 943569206 > Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa) > www.binovo.es > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
Hi Sinan, Just to reiterate: don't do this. Consumer SSDs will destroy your enterprise SSD's performance. Our office cluster is made of consumer-grade servers: cheap gaming motherboards, memory, ryzen processors, desktop HDDs. But SSD drives are Enterprise, we had awful experiences with consumer SSDs (some perform worse that HDDs with Ceph). Cheers Eneko El 19/12/19 a las 20:20, Sinan Polat escribió: Hi all, Thanks for the replies. I am not worried about their lifetime. We will be adding only 1 SSD disk per physical server. All SSD’s are enterprise drives. If the added consumer grade disk will fail, no problem. I am more curious regarding their I/O performance. I do want to have 50% drop in performance. So anyone any experience with 860 EVO or Crucial MX500 in a Ceph setup? Thanks! Op 19 dec. 2019 om 19:18 heeft Mark Nelson het volgende geschreven: The way I try to look at this is: 1) How much more do the enterprise grade drives cost? 2) What are the benefits? (Faster performance, longer life, etc) 3) How much does it cost to deal with downtime, diagnose issues, and replace malfunctioning hardware? My personal take is that enterprise drives are usually worth it. There may be consumer grade drives that may be worth considering in very specific scenarios if they still have power loss protection and high write durability. Even when I was in academia years ago with very limited budgets, we got burned with consumer grade SSDs to the point where we had to replace them all. You have to be very careful and know exactly what you are buying. Mark On 12/19/19 12:04 PM, jes...@krogh.cc wrote: I dont think “usually” is good enough in a production setup. Sent from myMail for iOS Thursday, 19 December 2019, 12.09 +0100 from Виталий Филиппов : Usually it doesn't, it only harms performance and probably SSD lifetime too > I would not be running ceph on ssds without powerloss protection. I > delivers a potential data loss scenario ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Zuzendari Teknikoa / Director Técnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
Hi Sinan, I would not recommend using 860 EVO or Crucial MX500 SSD's in a Ceph cluster, as those are consumer grade solutions and not enterprise ones. Performance and durability will be issues. If feasible, I would simply go NVMe as it sounds like you will be using this disk to store the journal or db partition. From: ceph-users on behalf of Antoine Lecrux Sent: Thursday, December 19, 2019 4:02 PM To: Udo Lembke ; Sinan Polat Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Consumer-grade SSD in Ceph Hi, If you're looking for a consumer grade SSD, make sure it has capacitors to protect you from data corruption in case of a power outage on the entire Ceph Cluster. That's the most important technical specification to look for. - Antoine -Original Message- From: ceph-users On Behalf Of Udo Lembke Sent: Thursday, December 19, 2019 3:22 PM To: Sinan Polat Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Consumer-grade SSD in Ceph Hi, if you add on more than one server an SSD with an short lifetime, you can run in real trouble (dataloss)! Even if, all other SSDs are enterprise grade. Ceph mix all data in PGs, which are spread over many disks - if one disk fails - no poblem, but if the next two fails after that due high io (recovery) you will have data loss. But if you have only one node with consumer SSDs, the whole node can go down without trouble... I've tried consumer SSDs as yournal a long time ago - was an bad idea! But this SSDs are cheap - buy one and do the io-test. If you monitoring the live-time it's perhaps possible for your setup. Udo Am 19.12.19 um 20:20 schrieb Sinan Polat: > Hi all, > > Thanks for the replies. I am not worried about their lifetime. We will be > adding only 1 SSD disk per physical server. All SSD’s are enterprise drives. > If the added consumer grade disk will fail, no problem. > > I am more curious regarding their I/O performance. I do want to have 50% drop > in performance. > > So anyone any experience with 860 EVO or Crucial MX500 in a Ceph setup? > > Thanks! > >> Op 19 dec. 2019 om 19:18 heeft Mark Nelson het volgende >> geschreven: >> >> The way I try to look at this is: >> >> >> 1) How much more do the enterprise grade drives cost? >> >> 2) What are the benefits? (Faster performance, longer life, etc) >> >> 3) How much does it cost to deal with downtime, diagnose issues, and replace >> malfunctioning hardware? >> >> >> My personal take is that enterprise drives are usually worth it. There may >> be consumer grade drives that may be worth considering in very specific >> scenarios if they still have power loss protection and high write >> durability. Even when I was in academia years ago with very limited >> budgets, we got burned with consumer grade SSDs to the point where we had to >> replace them all. You have to be very careful and know exactly what you are >> buying. >> >> >> Mark >> >> >>> On 12/19/19 12:04 PM, jes...@krogh.cc wrote: >>> I dont think “usually” is good enough in a production setup. >>> >>> >>> >>> Sent from myMail for iOS >>> >>> >>> Thursday, 19 December 2019, 12.09 +0100 from Виталий Филиппов >>> : >>> >>>Usually it doesn't, it only harms performance and probably SSD >>>lifetime >>>too >>> >>>> I would not be running ceph on ssds without powerloss protection. I >>>> delivers a potential data loss scenario >>> >>> >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
I had. 100-200 write iops with iodepth=1, ~5k iops with iodepth=128. These were intel 545s. Not that awful, but micron 5200 costs only a fraction more, so it seems pointless to me to use desktop samsungs. 19 декабря 2019 г. 22:20:28 GMT+03:00, Sinan Polat пишет: >Hi all, > >Thanks for the replies. I am not worried about their lifetime. We will >be adding only 1 SSD disk per physical server. All SSD’s are enterprise >drives. If the added consumer grade disk will fail, no problem. > >I am more curious regarding their I/O performance. I do want to have >50% drop in performance. > >So anyone any experience with 860 EVO or Crucial MX500 in a Ceph setup? > >Thanks! > >> Op 19 dec. 2019 om 19:18 heeft Mark Nelson het >volgende geschreven: >> >> The way I try to look at this is: >> >> >> 1) How much more do the enterprise grade drives cost? >> >> 2) What are the benefits? (Faster performance, longer life, etc) >> >> 3) How much does it cost to deal with downtime, diagnose issues, and >replace malfunctioning hardware? >> >> >> My personal take is that enterprise drives are usually worth it. >There may be consumer grade drives that may be worth considering in >very specific scenarios if they still have power loss protection and >high write durability. Even when I was in academia years ago with very >limited budgets, we got burned with consumer grade SSDs to the point >where we had to replace them all. You have to be very careful and know >exactly what you are buying. >> >> >> Mark >> >> >>> On 12/19/19 12:04 PM, jes...@krogh.cc wrote: >>> I dont think “usually” is good enough in a production setup. >>> >>> >>> >>> Sent from myMail for iOS >>> >>> >>> Thursday, 19 December 2019, 12.09 +0100 from Виталий Филиппов >: >>> >>>Usually it doesn't, it only harms performance and probably SSD >>>lifetime >>>too >>> >>>> I would not be running ceph on ssds without powerloss >protection. I >>>> delivers a potential data loss scenario >>> >>> >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >___ >ceph-users mailing list >ceph-users@lists.ceph.com >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- With best regards, Vitaliy Filippov___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
Hi, If you're looking for a consumer grade SSD, make sure it has capacitors to protect you from data corruption in case of a power outage on the entire Ceph Cluster. That's the most important technical specification to look for. - Antoine -Original Message- From: ceph-users On Behalf Of Udo Lembke Sent: Thursday, December 19, 2019 3:22 PM To: Sinan Polat Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Consumer-grade SSD in Ceph Hi, if you add on more than one server an SSD with an short lifetime, you can run in real trouble (dataloss)! Even if, all other SSDs are enterprise grade. Ceph mix all data in PGs, which are spread over many disks - if one disk fails - no poblem, but if the next two fails after that due high io (recovery) you will have data loss. But if you have only one node with consumer SSDs, the whole node can go down without trouble... I've tried consumer SSDs as yournal a long time ago - was an bad idea! But this SSDs are cheap - buy one and do the io-test. If you monitoring the live-time it's perhaps possible for your setup. Udo Am 19.12.19 um 20:20 schrieb Sinan Polat: > Hi all, > > Thanks for the replies. I am not worried about their lifetime. We will be > adding only 1 SSD disk per physical server. All SSD’s are enterprise drives. > If the added consumer grade disk will fail, no problem. > > I am more curious regarding their I/O performance. I do want to have 50% drop > in performance. > > So anyone any experience with 860 EVO or Crucial MX500 in a Ceph setup? > > Thanks! > >> Op 19 dec. 2019 om 19:18 heeft Mark Nelson het volgende >> geschreven: >> >> The way I try to look at this is: >> >> >> 1) How much more do the enterprise grade drives cost? >> >> 2) What are the benefits? (Faster performance, longer life, etc) >> >> 3) How much does it cost to deal with downtime, diagnose issues, and replace >> malfunctioning hardware? >> >> >> My personal take is that enterprise drives are usually worth it. There may >> be consumer grade drives that may be worth considering in very specific >> scenarios if they still have power loss protection and high write >> durability. Even when I was in academia years ago with very limited >> budgets, we got burned with consumer grade SSDs to the point where we had to >> replace them all. You have to be very careful and know exactly what you are >> buying. >> >> >> Mark >> >> >>> On 12/19/19 12:04 PM, jes...@krogh.cc wrote: >>> I dont think “usually” is good enough in a production setup. >>> >>> >>> >>> Sent from myMail for iOS >>> >>> >>> Thursday, 19 December 2019, 12.09 +0100 from Виталий Филиппов >>> : >>> >>>Usually it doesn't, it only harms performance and probably SSD >>>lifetime >>>too >>> >>>> I would not be running ceph on ssds without powerloss protection. I >>>> delivers a potential data loss scenario >>> >>> >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
Hi, if you add on more than one server an SSD with an short lifetime, you can run in real trouble (dataloss)! Even if, all other SSDs are enterprise grade. Ceph mix all data in PGs, which are spread over many disks - if one disk fails - no poblem, but if the next two fails after that due high io (recovery) you will have data loss. But if you have only one node with consumer SSDs, the whole node can go down without trouble... I've tried consumer SSDs as yournal a long time ago - was an bad idea! But this SSDs are cheap - buy one and do the io-test. If you monitoring the live-time it's perhaps possible for your setup. Udo Am 19.12.19 um 20:20 schrieb Sinan Polat: > Hi all, > > Thanks for the replies. I am not worried about their lifetime. We will be > adding only 1 SSD disk per physical server. All SSD’s are enterprise drives. > If the added consumer grade disk will fail, no problem. > > I am more curious regarding their I/O performance. I do want to have 50% drop > in performance. > > So anyone any experience with 860 EVO or Crucial MX500 in a Ceph setup? > > Thanks! > >> Op 19 dec. 2019 om 19:18 heeft Mark Nelson het volgende >> geschreven: >> >> The way I try to look at this is: >> >> >> 1) How much more do the enterprise grade drives cost? >> >> 2) What are the benefits? (Faster performance, longer life, etc) >> >> 3) How much does it cost to deal with downtime, diagnose issues, and replace >> malfunctioning hardware? >> >> >> My personal take is that enterprise drives are usually worth it. There may >> be consumer grade drives that may be worth considering in very specific >> scenarios if they still have power loss protection and high write >> durability. Even when I was in academia years ago with very limited >> budgets, we got burned with consumer grade SSDs to the point where we had to >> replace them all. You have to be very careful and know exactly what you are >> buying. >> >> >> Mark >> >> >>> On 12/19/19 12:04 PM, jes...@krogh.cc wrote: >>> I dont think “usually” is good enough in a production setup. >>> >>> >>> >>> Sent from myMail for iOS >>> >>> >>> Thursday, 19 December 2019, 12.09 +0100 from Виталий Филиппов >>> : >>> >>>Usually it doesn't, it only harms performance and probably SSD >>>lifetime >>>too >>> >>>> I would not be running ceph on ssds without powerloss protection. I >>>> delivers a potential data loss scenario >>> >>> >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
Hi all, Thanks for the replies. I am not worried about their lifetime. We will be adding only 1 SSD disk per physical server. All SSD’s are enterprise drives. If the added consumer grade disk will fail, no problem. I am more curious regarding their I/O performance. I do want to have 50% drop in performance. So anyone any experience with 860 EVO or Crucial MX500 in a Ceph setup? Thanks! > Op 19 dec. 2019 om 19:18 heeft Mark Nelson het volgende > geschreven: > > The way I try to look at this is: > > > 1) How much more do the enterprise grade drives cost? > > 2) What are the benefits? (Faster performance, longer life, etc) > > 3) How much does it cost to deal with downtime, diagnose issues, and replace > malfunctioning hardware? > > > My personal take is that enterprise drives are usually worth it. There may be > consumer grade drives that may be worth considering in very specific > scenarios if they still have power loss protection and high write durability. > Even when I was in academia years ago with very limited budgets, we got > burned with consumer grade SSDs to the point where we had to replace them > all. You have to be very careful and know exactly what you are buying. > > > Mark > > >> On 12/19/19 12:04 PM, jes...@krogh.cc wrote: >> I dont think “usually” is good enough in a production setup. >> >> >> >> Sent from myMail for iOS >> >> >> Thursday, 19 December 2019, 12.09 +0100 from Виталий Филиппов >> : >> >>Usually it doesn't, it only harms performance and probably SSD >>lifetime >>too >> >>> I would not be running ceph on ssds without powerloss protection. I >>> delivers a potential data loss scenario >> >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
The way I try to look at this is: 1) How much more do the enterprise grade drives cost? 2) What are the benefits? (Faster performance, longer life, etc) 3) How much does it cost to deal with downtime, diagnose issues, and replace malfunctioning hardware? My personal take is that enterprise drives are usually worth it. There may be consumer grade drives that may be worth considering in very specific scenarios if they still have power loss protection and high write durability. Even when I was in academia years ago with very limited budgets, we got burned with consumer grade SSDs to the point where we had to replace them all. You have to be very careful and know exactly what you are buying. Mark On 12/19/19 12:04 PM, jes...@krogh.cc wrote: I dont think “usually” is good enough in a production setup. Sent from myMail for iOS Thursday, 19 December 2019, 12.09 +0100 from Виталий Филиппов : Usually it doesn't, it only harms performance and probably SSD lifetime too > I would not be running ceph on ssds without powerloss protection. I > delivers a potential data loss scenario ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
I dont think “usually” is good enough in a production setup. Sent from myMail for iOS Thursday, 19 December 2019, 12.09 +0100 from Виталий Филиппов : >Usually it doesn't, it only harms performance and probably SSD lifetime >too > >> I would not be running ceph on ssds without powerloss protection. I >> delivers a potential data loss scenario ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
Usually it doesn't, it only harms performance and probably SSD lifetime too I would not be running ceph on ssds without powerloss protection. I delivers a potential data loss scenario ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
I would not be running ceph on ssds without powerloss protection. I delivers a potential data loss scenario Jesper Sent from myMail for iOS Thursday, 19 December 2019, 08.32 +0100 from Виталий Филиппов : >https://yourcmc.ru/wiki/Ceph_performance > >https://docs.google.com/spreadsheets/d/1E9-eXjzsKboiCCX-0u0r5fAjjufLKayaut_FOPxYZjc > >19 декабря 2019 г. 0:41:02 GMT+03:00, Sinan Polat < si...@turka.nl > пишет: >>Hi, >> >>I am aware that >>https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ >> holds a list with benchmark of quite some different ssd models. >>Unfortunately it doesn't have benchmarks for recent ssd models. >> >>A client is planning to expand a running cluster (Luminous, FileStore, SSD >>only, Replicated). I/O Utilization is close to 0, but capacity wise the >>cluster is almost nearfull. To save costs the cluster will be expanded will >>customer-grade SSD's, but I am unable to find benchmarks of recent SSD models. >> >>Does anyone has experience with Samsung 860 EVO, 860 PRO and Crucial MX500 in >>a Ceph cluster? >> >>Thanks! >>Sinan >-- >With best regards, >Vitaliy Filippov >___ >ceph-users mailing list >ceph-users@lists.ceph.com >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
https://yourcmc.ru/wiki/Ceph_performance https://docs.google.com/spreadsheets/d/1E9-eXjzsKboiCCX-0u0r5fAjjufLKayaut_FOPxYZjc 19 декабря 2019 г. 0:41:02 GMT+03:00, Sinan Polat пишет: >Hi, > >I am aware that >https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ >holds a list with benchmark of quite some different ssd models. >Unfortunately it >doesn't have benchmarks for recent ssd models. > >A client is planning to expand a running cluster (Luminous, FileStore, >SSD only, >Replicated). I/O Utilization is close to 0, but capacity wise the >cluster is >almost nearfull. To save costs the cluster will be expanded will >customer-grade >SSD's, but I am unable to find benchmarks of recent SSD models. > >Does anyone has experience with Samsung 860 EVO, 860 PRO and Crucial >MX500 in a >Ceph cluster? > >Thanks! >Sinan -- With best regards, Vitaliy Filippov___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Consumer-grade SSD in Ceph
I know not your question, but what is the current SSD's your using in the cluster? You could end up with an imbalanced cluster performance wise, if you have a mix of SSD's with a large performance difference, specially with filestore and it's journal writes. On Thu, 19 Dec 2019 05:41:02 +0800 Sinan Polat wrote Hi, I am aware that https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ holds a list with benchmark of quite some different ssd models. Unfortunately it doesn't have benchmarks for recent ssd models. A client is planning to expand a running cluster (Luminous, FileStore, SSD only, Replicated). I/O Utilization is close to 0, but capacity wise the cluster is almost nearfull. To save costs the cluster will be expanded will customer-grade SSD's, but I am unable to find benchmarks of recent SSD models. Does anyone has experience with Samsung 860 EVO, 860 PRO and Crucial MX500 in a Ceph cluster? Thanks! Sinan ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Consumer-grade SSD in Ceph
Hi, I am aware that https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ holds a list with benchmark of quite some different ssd models. Unfortunately it doesn't have benchmarks for recent ssd models. A client is planning to expand a running cluster (Luminous, FileStore, SSD only, Replicated). I/O Utilization is close to 0, but capacity wise the cluster is almost nearfull. To save costs the cluster will be expanded will customer-grade SSD's, but I am unable to find benchmarks of recent SSD models. Does anyone has experience with Samsung 860 EVO, 860 PRO and Crucial MX500 in a Ceph cluster? Thanks! Sinan___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
