On Fri, 19 Sep 2014, Alexandre DERUMIER wrote: > >> with rbd_cache=true , I got around 60000iops (and I don't see any network > >> traffic) > >> > >>So maybe they are a bug in fio ? > >>maybe this is related to: > > Oh, sorry, this was my fault, I didn't fill the rbd with datas before doing > the bench > > Now the results are (for 1 osd) > > firefly > ------ > bw=37460KB/s, iops=9364 > > giant > ----- > bw=32741KB/s, iops=8185 > > > So, a little regression > > (the results are equals rbd_cache=true|false)
Do you see a difference with rados bench, or is it just librbd? Thanks! sage > > > I'll try to compare with more osds > > ----- Mail original ----- > > De: "Alexandre DERUMIER" <[email protected]> > ?: "Somnath Roy" <[email protected]> > Cc: "Sage Weil" <[email protected]>, "Josh Durgin" <[email protected]>, > [email protected], "Haomai Wang" <[email protected]> > Envoy?: Vendredi 19 Septembre 2014 12:09:41 > Objet: Re: severe librbd performance degradation in Giant > > >>What tool are you using ? I used fio rbd. > > fio rbd too > > > [global] > ioengine=rbd > clientname=admin > pool=test > rbdname=test > invalidate=0 > #rw=read > #rw=randwrite > #rw=write > rw=randread > bs=4k > direct=1 > numjobs=2 > group_reporting=1 > size=10G > > [rbd_iodepth32] > iodepth=32 > > > > I just notice something strange > > with rbd_cache=true , I got around 60000iops (and I don't see any network > traffic) > > So maybe they are a bug in fio ? > maybe this is related to: > > > http://tracker.ceph.com/issues/9391 > "fio rbd driver rewrites same blocks" > > ----- Mail original ----- > > De: "Somnath Roy" <[email protected]> > ?: "Alexandre DERUMIER" <[email protected]>, "Haomai Wang" > <[email protected]> > Cc: "Sage Weil" <[email protected]>, "Josh Durgin" <[email protected]>, > [email protected] > Envoy?: Jeudi 18 Septembre 2014 20:02:49 > Objet: RE: severe librbd performance degradation in Giant > > Alexandre, > What tool are you using ? I used fio rbd. > > Also, I hope you have Giant package installed in the client side as well and > rbd_cache =true is set on the client conf file. > FYI, firefly librbd + librados and Giant cluster will work seamlessly and I > had to make sure fio rbd is really loading giant librbd (if you have multiple > copies around , which was in my case) for reproducing it. > > Thanks & Regards > Somnath > > -----Original Message----- > From: Alexandre DERUMIER [mailto:[email protected]] > Sent: Thursday, September 18, 2014 2:49 AM > To: Haomai Wang > Cc: Sage Weil; Josh Durgin; [email protected]; Somnath Roy > Subject: Re: severe librbd performance degradation in Giant > > >>According http://tracker.ceph.com/issues/9513, do you mean that rbd > >>cache will make 10x performance degradation for random read? > > Hi, on my side, I don't see any degradation performance on read (seq or rand) > with or without. > > firefly : around 12000iops (with or without rbd_cache) giant : around > 12000iops (with or without rbd_cache) > > (and I can reach around 20000-30000 iops on giant with disabling optracker). > > > rbd_cache only improve write performance for me (4k block ) > > > > ----- Mail original ----- > > De: "Haomai Wang" <[email protected]> > ?: "Somnath Roy" <[email protected]> > Cc: "Sage Weil" <[email protected]>, "Josh Durgin" <[email protected]>, > [email protected] > Envoy?: Jeudi 18 Septembre 2014 04:27:56 > Objet: Re: severe librbd performance degradation in Giant > > According http://tracker.ceph.com/issues/9513, do you mean that rbd cache > will make 10x performance degradation for random read? > > On Thu, Sep 18, 2014 at 7:44 AM, Somnath Roy <[email protected]> wrote: > > Josh/Sage, > > I should mention that even after turning off rbd cache I am getting ~20% > > degradation over Firefly. > > > > Thanks & Regards > > Somnath > > > > -----Original Message----- > > From: Somnath Roy > > Sent: Wednesday, September 17, 2014 2:44 PM > > To: Sage Weil > > Cc: Josh Durgin; [email protected] > > Subject: RE: severe librbd performance degradation in Giant > > > > Created a tracker for this. > > > > http://tracker.ceph.com/issues/9513 > > > > Thanks & Regards > > Somnath > > > > -----Original Message----- > > From: [email protected] > > [mailto:[email protected]] On Behalf Of Somnath Roy > > Sent: Wednesday, September 17, 2014 2:39 PM > > To: Sage Weil > > Cc: Josh Durgin; [email protected] > > Subject: RE: severe librbd performance degradation in Giant > > > > Sage, > > It's a 4K random read. > > > > Thanks & Regards > > Somnath > > > > -----Original Message----- > > From: Sage Weil [mailto:[email protected]] > > Sent: Wednesday, September 17, 2014 2:36 PM > > To: Somnath Roy > > Cc: Josh Durgin; [email protected] > > Subject: RE: severe librbd performance degradation in Giant > > > > What was the io pattern? Sequential or random? For random a slowdown makes > > sense (tho maybe not 10x!) but not for sequentail.... > > > > s > > > > On Wed, 17 Sep 2014, Somnath Roy wrote: > > > >> I set the following in the client side /etc/ceph/ceph.conf where I am > >> running fio rbd. > >> > >> rbd_cache_writethrough_until_flush = false > >> > >> But, no difference. BTW, I am doing Random read, not write. Still this > >> setting applies ? > >> > >> Next, I tried to tweak the rbd_cache setting to false and I *got back* the > >> old performance. Now, it is similar to firefly throughput ! > >> > >> So, loks like rbd_cache=true was the culprit. > >> > >> Thanks Josh ! > >> > >> Regards > >> Somnath > >> > >> -----Original Message----- > >> From: Josh Durgin [mailto:[email protected]] > >> Sent: Wednesday, September 17, 2014 2:20 PM > >> To: Somnath Roy; [email protected] > >> Subject: Re: severe librbd performance degradation in Giant > >> > >> On 09/17/2014 01:55 PM, Somnath Roy wrote: > >> > Hi Sage, > >> > We are experiencing severe librbd performance degradation in Giant over > >> > firefly release. Here is the experiment we did to isolate it as a librbd > >> > problem. > >> > > >> > 1. Single OSD is running latest Giant and client is running fio rbd on > >> > top of firefly based librbd/librados. For one client it is giving > >> > ~11-12K iops (4K RR). > >> > 2. Single OSD is running Giant and client is running fio rbd on top of > >> > Giant based librbd/librados. For one client it is giving ~1.9K iops (4K > >> > RR). > >> > 3. Single OSD is running latest Giant and client is running Giant based > >> > ceph_smaiobench on top of giant librados. For one client it is giving > >> > ~11-12K iops (4K RR). > >> > 4. Giant RGW on top of Giant OSD is also scaling. > >> > > >> > > >> > So, it is obvious from the above that recent librbd has issues. I will > >> > raise a tracker to track this. > >> > >> For giant the default cache settings changed to: > >> > >> rbd cache = true > >> rbd cache writethrough until flush = true > >> > >> If fio isn't sending flushes as the test is running, the cache will stay > >> in writethrough mode. Does the difference remain if you set rbd cache > >> writethrough until flush = false ? > >> > >> Josh > >> > >> ________________________________ > >> > >> PLEASE NOTE: The information contained in this electronic mail message is > >> intended only for the use of the designated recipient(s) named above. If > >> the reader of this message is not the intended recipient, you are hereby > >> notified that you have received this message in error and that any review, > >> dissemination, distribution, or copying of this message is strictly > >> prohibited. If you have received this communication in error, please > >> notify the sender by telephone or e-mail (as shown above) immediately and > >> destroy any and all copies of this message in your possession (whether > >> hard copies or electronically stored copies). > >> > >> -- > >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" > >> in the body of a message to [email protected] More majordomo > >> info at http://vger.kernel.org/majordomo-info.html > >> > >> > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > in the body of a message to [email protected] More majordomo > > info at http://vger.kernel.org/majordomo-info.html > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > in the body of a message to [email protected] More majordomo > > info at http://vger.kernel.org/majordomo-info.html > > > > -- > Best Regards, > > Wheat > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the > body of a message to [email protected] More majordomo info at > http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to [email protected] > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to [email protected] > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
