Me and quite a few others have had high random latency issues with disk cache enabled.
,Ash On Tue, 20 Nov 2018 at 9:09 PM, Alex Litvak <[email protected]> wrote: > John, > > If I go with write through, shouldn't disk cache be enabled? > > On 11/20/2018 6:12 AM, John Petrini wrote: > > I would disable cache on the controller for your journals. Use write > through and no read ahead. Did you make sure the disk cache is disabled? > > > > On Tuesday, November 20, 2018, Alex Litvak <[email protected] > <mailto:[email protected]>> wrote: > > > I went through raid controller firmware update. I replaced a pair > of SSDs with new ones. Nothing have changed. Per controller card utility > it shows that no patrol reading happens and battery > > backup is in a good shape. Cache policy is WriteBack. I am aware on > the bad battery effect but it doesn't seem to be the case unless controller > is lying to me. > > > > > > > > > On 11/19/2018 2:39 PM, Brendan Moloney wrote: > > >> > > >> Hi, > > >> > > >>> Raid card for journal disks is Perc H730 (Megaraid), RAID 1, > battery back cache is on > > >>> > > >>> Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write > Cache if Bad BBU > > >>> Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write > Cache if Bad BBU > > >>> > > >>> I have 2 other nodes with older Perc H710 and similar SSDs with > slightly higher wear (6.3% vs 5.18%) but from observation they hardly hit > 1.5 ms on rear occasion > > >>> Cache, RAID, and battery situation is the same. > > >> > > >> I would take a closer look at the RAID card. Are you sure the BBU > is ok? In the past I noticed the Megaraid cards would do periodic battery > tests that would completely drain the battery and thus > > disable the write cache until they reached some threshold of charge > again. They also can do periodic "patrol reads" and "consistency checks" > that can hurt performance. Or the card could just be > > failing, I have almost gone through more RAID cards than HDDs. The > unreliability and black box nature of hardware RAID cards is one of the > things that first got me looking into Ceph (although even > > mdadm is a big improvement in my opinion). > > >> > > >> For journals you are better off putting half your OSDs on one SSD > and half on the other instead of RAID1. > > >> > > >> -Brendan > > >> > > > > > > > > > _______________________________________________ > > > ceph-users mailing list > > > [email protected] <mailto:[email protected]> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > -- > > > > John Petrini > > Platforms Engineer > > > > > > Call CoreDial 215.297.4400 x 232 <tel:215-297-4400> > > > > Call CoreDial www.coredial.com <https://coredial.com/> > > > > CoreDial 751 Arbor Way, Hillcrest I, Suite 150 Blue Bell, PA 19422 > > < > https://www.google.com/maps/place/CoreDial,+LLC/@40.140902,-75.2878857,17z/data=%213m1%214b1%214m5%213m4%211s0x89c6bc587f1cfd47:0x4c79d505f2ee580b%218m2%213d40.140902%214d-75.285697 > > > > > > The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. Any review, retransmission, dissemination or > > other use of, or taking of any action in reliance upon, this information > by persons or entities other than the intended recipient is prohibited. If > you received this in error, please contact the > > sender and delete the material from any computer. > > > > > > > > > > _______________________________________________ > > ceph-users mailing list > > [email protected] > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
