>> with rbd_cache=true , I got around 60000iops (and I don't see any network >> traffic) >> >>So maybe they are a bug in fio ? >>maybe this is related to:
Oh, sorry, this was my fault, I didn't fill the rbd with datas before doing the bench Now the results are (for 1 osd) firefly ------ bw=37460KB/s, iops=9364 giant ----- bw=32741KB/s, iops=8185 So, a little regression (the results are equals rbd_cache=true|false) I'll try to compare with more osds ----- Mail original ----- De: "Alexandre DERUMIER" <[email protected]> À: "Somnath Roy" <[email protected]> Cc: "Sage Weil" <[email protected]>, "Josh Durgin" <[email protected]>, [email protected], "Haomai Wang" <[email protected]> Envoyé: Vendredi 19 Septembre 2014 12:09:41 Objet: Re: severe librbd performance degradation in Giant >>What tool are you using ? I used fio rbd. fio rbd too [global] ioengine=rbd clientname=admin pool=test rbdname=test invalidate=0 #rw=read #rw=randwrite #rw=write rw=randread bs=4k direct=1 numjobs=2 group_reporting=1 size=10G [rbd_iodepth32] iodepth=32 I just notice something strange with rbd_cache=true , I got around 60000iops (and I don't see any network traffic) So maybe they are a bug in fio ? maybe this is related to: http://tracker.ceph.com/issues/9391 "fio rbd driver rewrites same blocks" ----- Mail original ----- De: "Somnath Roy" <[email protected]> À: "Alexandre DERUMIER" <[email protected]>, "Haomai Wang" <[email protected]> Cc: "Sage Weil" <[email protected]>, "Josh Durgin" <[email protected]>, [email protected] Envoyé: Jeudi 18 Septembre 2014 20:02:49 Objet: RE: severe librbd performance degradation in Giant Alexandre, What tool are you using ? I used fio rbd. Also, I hope you have Giant package installed in the client side as well and rbd_cache =true is set on the client conf file. FYI, firefly librbd + librados and Giant cluster will work seamlessly and I had to make sure fio rbd is really loading giant librbd (if you have multiple copies around , which was in my case) for reproducing it. Thanks & Regards Somnath -----Original Message----- From: Alexandre DERUMIER [mailto:[email protected]] Sent: Thursday, September 18, 2014 2:49 AM To: Haomai Wang Cc: Sage Weil; Josh Durgin; [email protected]; Somnath Roy Subject: Re: severe librbd performance degradation in Giant >>According http://tracker.ceph.com/issues/9513, do you mean that rbd >>cache will make 10x performance degradation for random read? Hi, on my side, I don't see any degradation performance on read (seq or rand) with or without. firefly : around 12000iops (with or without rbd_cache) giant : around 12000iops (with or without rbd_cache) (and I can reach around 20000-30000 iops on giant with disabling optracker). rbd_cache only improve write performance for me (4k block ) ----- Mail original ----- De: "Haomai Wang" <[email protected]> À: "Somnath Roy" <[email protected]> Cc: "Sage Weil" <[email protected]>, "Josh Durgin" <[email protected]>, [email protected] Envoyé: Jeudi 18 Septembre 2014 04:27:56 Objet: Re: severe librbd performance degradation in Giant According http://tracker.ceph.com/issues/9513, do you mean that rbd cache will make 10x performance degradation for random read? On Thu, Sep 18, 2014 at 7:44 AM, Somnath Roy <[email protected]> wrote: > Josh/Sage, > I should mention that even after turning off rbd cache I am getting ~20% > degradation over Firefly. > > Thanks & Regards > Somnath > > -----Original Message----- > From: Somnath Roy > Sent: Wednesday, September 17, 2014 2:44 PM > To: Sage Weil > Cc: Josh Durgin; [email protected] > Subject: RE: severe librbd performance degradation in Giant > > Created a tracker for this. > > http://tracker.ceph.com/issues/9513 > > Thanks & Regards > Somnath > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Somnath Roy > Sent: Wednesday, September 17, 2014 2:39 PM > To: Sage Weil > Cc: Josh Durgin; [email protected] > Subject: RE: severe librbd performance degradation in Giant > > Sage, > It's a 4K random read. > > Thanks & Regards > Somnath > > -----Original Message----- > From: Sage Weil [mailto:[email protected]] > Sent: Wednesday, September 17, 2014 2:36 PM > To: Somnath Roy > Cc: Josh Durgin; [email protected] > Subject: RE: severe librbd performance degradation in Giant > > What was the io pattern? Sequential or random? For random a slowdown makes > sense (tho maybe not 10x!) but not for sequentail.... > > s > > On Wed, 17 Sep 2014, Somnath Roy wrote: > >> I set the following in the client side /etc/ceph/ceph.conf where I am >> running fio rbd. >> >> rbd_cache_writethrough_until_flush = false >> >> But, no difference. BTW, I am doing Random read, not write. Still this >> setting applies ? >> >> Next, I tried to tweak the rbd_cache setting to false and I *got back* the >> old performance. Now, it is similar to firefly throughput ! >> >> So, loks like rbd_cache=true was the culprit. >> >> Thanks Josh ! >> >> Regards >> Somnath >> >> -----Original Message----- >> From: Josh Durgin [mailto:[email protected]] >> Sent: Wednesday, September 17, 2014 2:20 PM >> To: Somnath Roy; [email protected] >> Subject: Re: severe librbd performance degradation in Giant >> >> On 09/17/2014 01:55 PM, Somnath Roy wrote: >> > Hi Sage, >> > We are experiencing severe librbd performance degradation in Giant over >> > firefly release. Here is the experiment we did to isolate it as a librbd >> > problem. >> > >> > 1. Single OSD is running latest Giant and client is running fio rbd on top >> > of firefly based librbd/librados. For one client it is giving ~11-12K iops >> > (4K RR). >> > 2. Single OSD is running Giant and client is running fio rbd on top of >> > Giant based librbd/librados. For one client it is giving ~1.9K iops (4K >> > RR). >> > 3. Single OSD is running latest Giant and client is running Giant based >> > ceph_smaiobench on top of giant librados. For one client it is giving >> > ~11-12K iops (4K RR). >> > 4. Giant RGW on top of Giant OSD is also scaling. >> > >> > >> > So, it is obvious from the above that recent librbd has issues. I will >> > raise a tracker to track this. >> >> For giant the default cache settings changed to: >> >> rbd cache = true >> rbd cache writethrough until flush = true >> >> If fio isn't sending flushes as the test is running, the cache will stay in >> writethrough mode. Does the difference remain if you set rbd cache >> writethrough until flush = false ? >> >> Josh >> >> ________________________________ >> >> PLEASE NOTE: The information contained in this electronic mail message is >> intended only for the use of the designated recipient(s) named above. If the >> reader of this message is not the intended recipient, you are hereby >> notified that you have received this message in error and that any review, >> dissemination, distribution, or copying of this message is strictly >> prohibited. If you have received this communication in error, please notify >> the sender by telephone or e-mail (as shown above) immediately and destroy >> any and all copies of this message in your possession (whether hard copies >> or electronically stored copies). >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >> in the body of a message to [email protected] More majordomo >> info at http://vger.kernel.org/majordomo-info.html >> >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to [email protected] More majordomo > info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to [email protected] More majordomo > info at http://vger.kernel.org/majordomo-info.html -- Best Regards, Wheat -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
