>>Crazy, I've 56 SSDs and canÄt go above 20 000 iops. I just notice than my fio benchmark is cpu bound...
I can reach around 40000iops. Don't have more client machines for the moment to bench ----- Mail original ----- De: "Stefan Priebe - Profihost AG" <[email protected]> À: "Xinxin Shu" <[email protected]>, "Somnath Roy" <[email protected]>, "Alexandre DERUMIER" <[email protected]>, "Haomai Wang" <[email protected]> Cc: "Sage Weil" <[email protected]>, "Josh Durgin" <[email protected]>, [email protected] Envoyé: Vendredi 19 Septembre 2014 15:31:14 Objet: Re: severe librbd performance degradation in Giant Am 19.09.2014 um 15:02 schrieb Shu, Xinxin: > 12 x Intel DC 3700 200GB, every SSD has two OSDs. Crazy, I've 56 SSDs and canÄt go above 20 000 iops. Grüße Stefan > Cheers, > xinxin > > -----Original Message----- > From: Stefan Priebe [mailto:[email protected]] > Sent: Friday, September 19, 2014 2:54 PM > To: Shu, Xinxin; Somnath Roy; Alexandre DERUMIER; Haomai Wang > Cc: Sage Weil; Josh Durgin; [email protected] > Subject: Re: severe librbd performance degradation in Giant > > Am 19.09.2014 03:08, schrieb Shu, Xinxin: >> I also observed performance degradation on my full SSD setup , I can >> got ~270K IOPS for 4KB random read with 0.80.4 , but with latest >> master , I only got ~12K IOPS > > This are impressive numbers. Can you tell me how many OSDs you have and which > SSDs you use? > > Thanks, > Stefan > > >> Cheers, >> xinxin >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of Somnath Roy >> Sent: Friday, September 19, 2014 2:03 AM >> To: Alexandre DERUMIER; Haomai Wang >> Cc: Sage Weil; Josh Durgin; [email protected] >> Subject: RE: severe librbd performance degradation in Giant >> >> Alexandre, >> What tool are you using ? I used fio rbd. >> >> Also, I hope you have Giant package installed in the client side as well and >> rbd_cache =true is set on the client conf file. >> FYI, firefly librbd + librados and Giant cluster will work seamlessly and I >> had to make sure fio rbd is really loading giant librbd (if you have >> multiple copies around , which was in my case) for reproducing it. >> >> Thanks & Regards >> Somnath >> >> -----Original Message----- >> From: Alexandre DERUMIER [mailto:[email protected]] >> Sent: Thursday, September 18, 2014 2:49 AM >> To: Haomai Wang >> Cc: Sage Weil; Josh Durgin; [email protected]; Somnath Roy >> Subject: Re: severe librbd performance degradation in Giant >> >>>> According http://tracker.ceph.com/issues/9513, do you mean that rbd >>>> cache will make 10x performance degradation for random read? >> >> Hi, on my side, I don't see any degradation performance on read (seq or >> rand) with or without. >> >> firefly : around 12000iops (with or without rbd_cache) giant : around >> 12000iops (with or without rbd_cache) >> >> (and I can reach around 20000-30000 iops on giant with disabling optracker). >> >> >> rbd_cache only improve write performance for me (4k block ) >> >> >> >> ----- Mail original ----- >> >> De: "Haomai Wang" <[email protected]> >> À: "Somnath Roy" <[email protected]> >> Cc: "Sage Weil" <[email protected]>, "Josh Durgin" >> <[email protected]>, [email protected] >> Envoyé: Jeudi 18 Septembre 2014 04:27:56 >> Objet: Re: severe librbd performance degradation in Giant >> >> According http://tracker.ceph.com/issues/9513, do you mean that rbd cache >> will make 10x performance degradation for random read? >> >> On Thu, Sep 18, 2014 at 7:44 AM, Somnath Roy <[email protected]> >> wrote: >>> Josh/Sage, >>> I should mention that even after turning off rbd cache I am getting ~20% >>> degradation over Firefly. >>> >>> Thanks & Regards >>> Somnath >>> >>> -----Original Message----- >>> From: Somnath Roy >>> Sent: Wednesday, September 17, 2014 2:44 PM >>> To: Sage Weil >>> Cc: Josh Durgin; [email protected] >>> Subject: RE: severe librbd performance degradation in Giant >>> >>> Created a tracker for this. >>> >>> http://tracker.ceph.com/issues/9513 >>> >>> Thanks & Regards >>> Somnath >>> >>> -----Original Message----- >>> From: [email protected] >>> [mailto:[email protected]] On Behalf Of Somnath Roy >>> Sent: Wednesday, September 17, 2014 2:39 PM >>> To: Sage Weil >>> Cc: Josh Durgin; [email protected] >>> Subject: RE: severe librbd performance degradation in Giant >>> >>> Sage, >>> It's a 4K random read. >>> >>> Thanks & Regards >>> Somnath >>> >>> -----Original Message----- >>> From: Sage Weil [mailto:[email protected]] >>> Sent: Wednesday, September 17, 2014 2:36 PM >>> To: Somnath Roy >>> Cc: Josh Durgin; [email protected] >>> Subject: RE: severe librbd performance degradation in Giant >>> >>> What was the io pattern? Sequential or random? For random a slowdown makes >>> sense (tho maybe not 10x!) but not for sequentail.... >>> >>> s >>> >>> On Wed, 17 Sep 2014, Somnath Roy wrote: >>> >>>> I set the following in the client side /etc/ceph/ceph.conf where I am >>>> running fio rbd. >>>> >>>> rbd_cache_writethrough_until_flush = false >>>> >>>> But, no difference. BTW, I am doing Random read, not write. Still this >>>> setting applies ? >>>> >>>> Next, I tried to tweak the rbd_cache setting to false and I *got back* the >>>> old performance. Now, it is similar to firefly throughput ! >>>> >>>> So, loks like rbd_cache=true was the culprit. >>>> >>>> Thanks Josh ! >>>> >>>> Regards >>>> Somnath >>>> >>>> -----Original Message----- >>>> From: Josh Durgin [mailto:[email protected]] >>>> Sent: Wednesday, September 17, 2014 2:20 PM >>>> To: Somnath Roy; [email protected] >>>> Subject: Re: severe librbd performance degradation in Giant >>>> >>>> On 09/17/2014 01:55 PM, Somnath Roy wrote: >>>>> Hi Sage, >>>>> We are experiencing severe librbd performance degradation in Giant over >>>>> firefly release. Here is the experiment we did to isolate it as a librbd >>>>> problem. >>>>> >>>>> 1. Single OSD is running latest Giant and client is running fio rbd on >>>>> top of firefly based librbd/librados. For one client it is giving ~11-12K >>>>> iops (4K RR). >>>>> 2. Single OSD is running Giant and client is running fio rbd on top of >>>>> Giant based librbd/librados. For one client it is giving ~1.9K iops (4K >>>>> RR). >>>>> 3. Single OSD is running latest Giant and client is running Giant based >>>>> ceph_smaiobench on top of giant librados. For one client it is giving >>>>> ~11-12K iops (4K RR). >>>>> 4. Giant RGW on top of Giant OSD is also scaling. >>>>> >>>>> >>>>> So, it is obvious from the above that recent librbd has issues. I will >>>>> raise a tracker to track this. >>>> >>>> For giant the default cache settings changed to: >>>> >>>> rbd cache = true >>>> rbd cache writethrough until flush = true >>>> >>>> If fio isn't sending flushes as the test is running, the cache will stay >>>> in writethrough mode. Does the difference remain if you set rbd cache >>>> writethrough until flush = false ? >>>> >>>> Josh >>>> >>>> ________________________________ >>>> >>>> PLEASE NOTE: The information contained in this electronic mail message is >>>> intended only for the use of the designated recipient(s) named above. If >>>> the reader of this message is not the intended recipient, you are hereby >>>> notified that you have received this message in error and that any review, >>>> dissemination, distribution, or copying of this message is strictly >>>> prohibited. If you have received this communication in error, please >>>> notify the sender by telephone or e-mail (as shown above) immediately and >>>> destroy any and all copies of this message in your possession (whether >>>> hard copies or electronically stored copies). >>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >>>> in the body of a message to [email protected] More majordomo >>>> info at http://vger.kernel.org/majordomo-info.html >>>> >>>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >>> in the body of a message to [email protected] More majordomo >>> info at http://vger.kernel.org/majordomo-info.html >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >>> in the body of a message to [email protected] More majordomo >>> info at http://vger.kernel.org/majordomo-info.html >> >> >> >> -- >> Best Regards, >> >> Wheat >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the >> body of a message to [email protected] More majordomo info at >> http://vger.kernel.org/majordomo-info.html >> N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w j:+v w j m zZ+ ݢj" ! i >> N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w > j:+v w j m zZ+ ݢj" !tml= >> > N�����r��y���b�X��ǧv�^�){.n�+���z�]z���{ay�ʇڙ�,j��f���h���z��w������j:+v���w�j�m��������zZ+�����ݢj"��!tml= > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
