Hi,
On 04/02/15 19:31, Stillwell, Bryan wrote:
> All,
>
> Whenever we're doing some kind of recovery operation on our ceph
> clusters (cluster expansion or dealing with a drive failure), there
> seems to be a fairly noticable performance drop while it does the
> backfills (last time I measured it the performance during recovery was
> something like 20% of a healthy cluster). I'm wondering if there are
> any settings that we might be missing which would improve this
> situation?
>
> Before doing any kind of expansion operation I make sure both 'noscrub'
> and 'nodeep-scrub' are set to make sure scrubing isn't making things
> worse.
>
> Also we have the following options set in our ceph.conf:
>
> [osd]
> osd_journal_size = 16384
> osd_max_backfills = 1
> osd_recovery_max_active = 1
> osd_recovery_op_priority = 1
> osd_recovery_max_single_start = 1
> osd_op_threads = 12
> osd_crush_initial_weight = 0
>
>
> I'm wondering if there might be a way to use ionice in the CFQ scheduler
> to delegate the recovery traffic to be of the Idle type so customer
> traffic has a higher priority?
Recovery creates I/O performance drops in our VM too but it's
manageable. What really hurts us are deep scrubs.
Our current situation is Firefly 0.80.9 with a total of 24 identical
OSDs evenly distributed on 4 servers with the following relevant
configuration:
osd recovery max active = 2
osd scrub load threshold = 3
osd deep scrub interval = 1209600 # 14 days
osd max backfills = 4
osd disk thread ioprio class = idle
osd disk thread ioprio priority = 7
we managed to add several OSDs at once while deep scrubs were in
practice disabled: we just increased deep scrub interval from 1 to 2
weeks which if I understand correctly had the effect of disabling them
for 1 week (and indeed there were none while the backfilling went on for
several hours).
With these settings and no deep-scrubs the load increased a bit in the
VMs doing non negligible I/Os but this was manageable. Even disk thread
ioprio settings (which is what you want to get the ionice behaviour for
deep scrubs) didn't seem to make much of a difference.
Note : I don't believe Ceph will try to scatter the scrubs on the whole
period you set with deep scrub interval, it seems its algorithm is much
simpler than that and may lead to temporary salves of successive deep
scrubs and it might generate some temporary I/O load which is hard to
diagnose (by default scrubs and deep scrubs are logged by the OSD so you
can correlate them with whatever you use to supervise your cluster).
I actually considered monitoring Ceph for backfills and using ceph set
nodeep-scrub automatically when there are some and unset it when they
disappear.
Best regards,
Lionel Bouton
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
