On Thu, 18 Sep 2014, Somnath Roy wrote: > Sage, > Any reason why the cache is by default enabled in Giant ?
It's recommended practice to turn it on. It improves performance in general (especially with HDD OSDs). Do you mind comparing sequential small IOs? sage > Regarding profiling, I will try if I can run Vtune/mutrace on this. > > Thanks & Regards > Somnath > > -----Original Message----- > From: Sage Weil [mailto:[email protected]] > Sent: Wednesday, September 17, 2014 8:53 PM > To: Somnath Roy > Cc: Haomai Wang; Josh Durgin; [email protected] > Subject: RE: severe librbd performance degradation in Giant > > On Thu, 18 Sep 2014, Somnath Roy wrote: > > Yes Haomai... > > I would love to what a profiler says about the matter. There is going to be > some overhead on the client associated with the cache for a random io > workload, but 10x is a problem! > > sage > > > > > > -----Original Message----- > > From: Haomai Wang [mailto:[email protected]] > > Sent: Wednesday, September 17, 2014 7:28 PM > > To: Somnath Roy > > Cc: Sage Weil; Josh Durgin; [email protected] > > Subject: Re: severe librbd performance degradation in Giant > > > > According http://tracker.ceph.com/issues/9513, do you mean that rbd cache > > will make 10x performance degradation for random read? > > > > On Thu, Sep 18, 2014 at 7:44 AM, Somnath Roy <[email protected]> > > wrote: > > > Josh/Sage, > > > I should mention that even after turning off rbd cache I am getting ~20% > > > degradation over Firefly. > > > > > > Thanks & Regards > > > Somnath > > > > > > -----Original Message----- > > > From: Somnath Roy > > > Sent: Wednesday, September 17, 2014 2:44 PM > > > To: Sage Weil > > > Cc: Josh Durgin; [email protected] > > > Subject: RE: severe librbd performance degradation in Giant > > > > > > Created a tracker for this. > > > > > > http://tracker.ceph.com/issues/9513 > > > > > > Thanks & Regards > > > Somnath > > > > > > -----Original Message----- > > > From: [email protected] > > > [mailto:[email protected]] On Behalf Of Somnath Roy > > > Sent: Wednesday, September 17, 2014 2:39 PM > > > To: Sage Weil > > > Cc: Josh Durgin; [email protected] > > > Subject: RE: severe librbd performance degradation in Giant > > > > > > Sage, > > > It's a 4K random read. > > > > > > Thanks & Regards > > > Somnath > > > > > > -----Original Message----- > > > From: Sage Weil [mailto:[email protected]] > > > Sent: Wednesday, September 17, 2014 2:36 PM > > > To: Somnath Roy > > > Cc: Josh Durgin; [email protected] > > > Subject: RE: severe librbd performance degradation in Giant > > > > > > What was the io pattern? Sequential or random? For random a slowdown > > > makes sense (tho maybe not 10x!) but not for sequentail.... > > > > > > s > > > > > > On Wed, 17 Sep 2014, Somnath Roy wrote: > > > > > >> I set the following in the client side /etc/ceph/ceph.conf where I am > > >> running fio rbd. > > >> > > >> rbd_cache_writethrough_until_flush = false > > >> > > >> But, no difference. BTW, I am doing Random read, not write. Still this > > >> setting applies ? > > >> > > >> Next, I tried to tweak the rbd_cache setting to false and I *got back* > > >> the old performance. Now, it is similar to firefly throughput ! > > >> > > >> So, loks like rbd_cache=true was the culprit. > > >> > > >> Thanks Josh ! > > >> > > >> Regards > > >> Somnath > > >> > > >> -----Original Message----- > > >> From: Josh Durgin [mailto:[email protected]] > > >> Sent: Wednesday, September 17, 2014 2:20 PM > > >> To: Somnath Roy; [email protected] > > >> Subject: Re: severe librbd performance degradation in Giant > > >> > > >> On 09/17/2014 01:55 PM, Somnath Roy wrote: > > >> > Hi Sage, > > >> > We are experiencing severe librbd performance degradation in Giant > > >> > over firefly release. Here is the experiment we did to isolate it as a > > >> > librbd problem. > > >> > > > >> > 1. Single OSD is running latest Giant and client is running fio rbd on > > >> > top of firefly based librbd/librados. For one client it is giving > > >> > ~11-12K iops (4K RR). > > >> > 2. Single OSD is running Giant and client is running fio rbd on top of > > >> > Giant based librbd/librados. For one client it is giving ~1.9K iops > > >> > (4K RR). > > >> > 3. Single OSD is running latest Giant and client is running Giant > > >> > based ceph_smaiobench on top of giant librados. For one client it is > > >> > giving ~11-12K iops (4K RR). > > >> > 4. Giant RGW on top of Giant OSD is also scaling. > > >> > > > >> > > > >> > So, it is obvious from the above that recent librbd has issues. I will > > >> > raise a tracker to track this. > > >> > > >> For giant the default cache settings changed to: > > >> > > >> rbd cache = true > > >> rbd cache writethrough until flush = true > > >> > > >> If fio isn't sending flushes as the test is running, the cache will stay > > >> in writethrough mode. Does the difference remain if you set rbd cache > > >> writethrough until flush = false ? > > >> > > >> Josh > > >> > > >> ________________________________ > > >> > > >> PLEASE NOTE: The information contained in this electronic mail message > > >> is intended only for the use of the designated recipient(s) named above. > > >> If the reader of this message is not the intended recipient, you are > > >> hereby notified that you have received this message in error and that > > >> any review, dissemination, distribution, or copying of this message is > > >> strictly prohibited. If you have received this communication in error, > > >> please notify the sender by telephone or e-mail (as shown above) > > >> immediately and destroy any and all copies of this message in your > > >> possession (whether hard copies or electronically stored copies). > > >> > > >> -- > > >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > >> in the body of a message to [email protected] More > > >> majordomo info at http://vger.kernel.org/majordomo-info.html > > >> > > >> > > > -- > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > > in the body of a message to [email protected] More majordomo > > > info at http://vger.kernel.org/majordomo-info.html > > > -- > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > > in the body of a message to [email protected] More majordomo > > > info at http://vger.kernel.org/majordomo-info.html > > > > > > > > -- > > Best Regards, > > > > Wheat > > N?????r??y??????X???v???)?{.n?????z?]z????ay? ????j ??f???h????? ?w??? > ???j:+v???w???????? ????zZ+???????j"????i > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to [email protected] > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
