Those are strange numbers, where are you getting them from? Test the drives directly with fio with every combination, that’s should tell you what’s happening
Jan > On 18 Jun 2015, at 07:52, Mateusz Skała <mateusz.sk...@budikom.net> wrote: > > Thanks for answer, > > I made some test, first leave dwc=enabled and caching on journal drive > disabled. Latency grows from 20ms to 90ms on this drive. Next I enabled cache > on journal drive and disabled all cache on data drives. Latency on data > drives grows from 30 – 50ms to 1500 – 2000ms. > Test made only on one osd host with P410i controller, with SATA drives > ST1000LM014-1EJ1 for data and for journal SSD INTEL SSDSC2BW12. > Regards, > Mateusz > > > From: Jan Schermer [mailto:j...@schermer.cz] > Sent: Wednesday, June 17, 2015 9:41 AM > To: Mateusz Skała > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] Hardware cache settings recomendation > > Cache on top of the data drives (not journal) will not help in most cases, > those writes are already buffered in the OS - so unless your OS is very light > on memory and flushing constantly it will have no effect, it just adds > overhead in case a flush comes. I haven’t tested this extensively with Ceph, > though. > > Cache enabled on journal drive _could_ help if your SSD is very slow (or if > you don’t have SSD for journal at all), and if it is large enough (more than > the active journal size) it could prolong the life of your SSD - depending on > how and when the cache starts to flush. I know from experience that write > cache on Areca controller didn't flush at all until it hit a watermark (50% > capacity default or something) and it will be faster than some SSDon their > own. Some SSD have higher IOPS than the cache can achieve, but you likely > won’t saturate that with Ceph. > > Another thing is write cache on the drives themselves - I’d leave that on > disabled (which is probably the default) unless the drive in question has > capacitors to flush the cache in case of power failure. Controllers usually > have a whitelist of devices that respect flushes on which the write cache is > default=enabled, but in case of for example Dell Perc you would need to have > Dell original drives or enable it manually. > > YMMV - i’ve hit the controller cache IOPS limit in the past with cheap Dell > Perc (H310 was it?) that did ~20K IOPS top on one SSD drive, while the drive > itself did close to 40K. On my SSDs, disabling write cache helps latency > (good for journal) bud could be troubling for the SSD lifetime. > > In any case I don’t think you would saturate either with Ceph, so I recommend > you just test the latency with write cache enabled/disabled on the controller > and pick the one that gives the best numbers > this is basically how: > http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ > > Ceph recommended way is to use everything as passthrough (initiator/target > mode) or JBOD (RAID0 with single drives on some controllers), so I’d stick > with that. > > Jan > > > On 17 Jun 2015, at 08:01, Mateusz Skała <mateusz.sk...@budikom.net> wrote: > > Yes, all disk are in single drive raid 0. Now cache is enabled for all > drives, should I disable cache for SSD drives? > Regards, > Mateusz > > From: Tyler Bishop [mailto:tyler.bis...@beyondhosting.net] > Sent: Thursday, June 11, 2015 7:30 PM > To: Mateusz Skała > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] Hardware cache settings recomendation > > You want write cache to disk, no write cache for SSD. > > I assume all of your data disk are single drive raid 0? > > > > Tyler Bishop > Chief Executive Officer > 513-299-7108 x10 > tyler.bis...@beyondhosting.net > > If you are not the intended recipient of this transmission you are notified > that disclosing, copying, distributing or taking any action in reliance on > the contents of this information is strictly prohibited. > > > ________________________________________ > From: "Mateusz Skała" <mateusz.sk...@budikom.net> > To: ceph-users@lists.ceph.com > Sent: Saturday, June 6, 2015 4:09:59 AM > Subject: [ceph-users] Hardware cache settings recomendation > > Hi, > Please help me with hardware cache settings on controllers for ceph rbd best > performance. All Ceph hosts have one SSD drive for journal. > > We are using 4 different controllers, all with BBU: > • HP Smart Array P400 > • HP Smart Array P410i > • Dell PERC 6/i > • Dell PERC H700 > > I have to set cache policy, on Dell settings are: > • Read Policy > o Read-Ahead (current) > o No-Read-Ahead > o Adaptive Read-Ahead > • Write Policy > o Write-Back (current) > o Write-Through > • Cache Policy > o Cache I/O > o Direct I/O (current) > • Disk Cache Policy > o Default (current) > o Enabled > o Disabled > On HP controllers: > • Cache Ratio (current: 25% Read / 75% Write) > • Drive Write Cache > o Enabled (current) > o Disabled > > And there is one more setting in LogicalDrive option: > • Caching: > o Enabled (current) > o Disabled > > Please verify my settings and give me some recomendations. > Best regards, > Mateusz > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
