Hi Simon
Do your workload has lots of RAW? Since Ceph has RW lock in each object,
so if you have a write to RBD and the following read happen to hit the same
object, the latency will be higher.
Another possibility is the OSD op_wq, it’s a priority queue but read and
write have same priority here and waiting Op_threads to serve them. So even
through you use deadline in the storage node ,but in general ,not making much
difference. Our BKM usually goes to set osd op threads to 20
If you could use ceph performance counter to identify if there is any
throttle has long wait time, together with your HW configuration and IOSTAT
data, that should be helpful for other to understand your case and help you.
Xiaoxi
From: ceph-users [mailto:[email protected]] On Behalf Of Nick
Fisk
Sent: Saturday, November 1, 2014 4:26 AM
To: [email protected]
Subject: Re: [ceph-users] prioritizing reads over writes
I think I might have to step out on this one, it sounds like you have all the
basics covered for best performance and I can’t think what else to suggest.
Sorry I couldn’t be of more help.
From: ceph-users [mailto:[email protected]] On Behalf Of Xu
(Simon) Chen
Sent: 31 October 2014 20:15
To: Nick Fisk
Cc: [email protected]<mailto:[email protected]>
Subject: Re: [ceph-users] prioritizing reads over writes
We have SSD journals, backend disks are actually on SSD-fronted bcache devices
in writeback mode. The client VMs have rbd cache enabled too...
-Simon
On Fri, Oct 31, 2014 at 4:07 PM, Nick Fisk
<[email protected]<mailto:[email protected]>> wrote:
Hmmm, it sounds like you are just saturating the spindles to the point that
latency starts to climb to unacceptable levels. The problem being that no
matter how much tuning you apply, at some point the writes will have to start
being put down to the disk and at that point performance will suffer.
Do your OSD’s have SSD journals? In storage, normally adding some sort of
writeback cache (in Ceph’s case Journals) help to lessen the impact of writes
by asorbing bursts of writes and by coalescing writes into a more sequential
pattern to the underlying disks.
From: ceph-users
[mailto:[email protected]<mailto:[email protected]>]
On Behalf Of Xu (Simon) Chen
Sent: 31 October 2014 19:51
To: Nick Fisk
Cc: [email protected]<mailto:[email protected]>
Subject: Re: [ceph-users] prioritizing reads over writes
I am already using deadline scheduler, with the default parameters:
read_expire=500
write_expire=5000
writes_starved=2
front_merges=1
fifo_batch=16
I remember tuning them before, didn't make a great difference.
-Simon
On Fri, Oct 31, 2014 at 3:43 PM, Nick Fisk
<[email protected]<mailto:[email protected]>> wrote:
Hi Simon,
Have you tried using the Deadline scheduler on the Linux nodes? The deadline
scheduler prioritises reads over writes. I believe it tries to service all
reads within 500ms whilst writes can be delayed up to 5s.
I don’t the exact effect Ceph will have over the top of this, but this would be
the first thing I would try.
Nick
From: ceph-users
[mailto:[email protected]<mailto:[email protected]>]
On Behalf Of Xu (Simon) Chen
Sent: 31 October 2014 19:37
To: [email protected]<mailto:[email protected]>
Subject: [ceph-users] prioritizing reads over writes
Hi all,
My workload is mostly writes, but when the writes reach a certain throughput
(iops wise not much higher) the read throughput would tank. This seems to be
impacting my VMs' responsiveness overall. Reads would recover after write
throughput drops.
Is there any way to prioritize read over write, or at least guarantee a certain
level of aggregated read throughput in a cluster?
Thanks.
-Simon
_______________________________________________
ceph-users mailing list
[email protected]<mailto:[email protected]>
http://xo4t.mjt.lu/link/xo4t/tsn3tlt/1/J2miz1ulcJHX7Mrh6NXZdQ/aHR0cDovL2xpc3RzLmNlcGguY29tL2xpc3RpbmZvLmNnaS9jZXBoLXVzZXJzLWNlcGguY29t
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
