On 09/10/2015 10:56 PM, Robert LeBlanc wrote: > Things I've tried: > > * Lowered nr_requests on the spindles from 1000 to 100. This reduced > the max latency sometimes up to 3000 ms down to a max of 500-700 ms. > it has also reduced the huge swings in latency, but has also reduced > throughput somewhat. > * Changed the scheduler from deadline to CFQ. I'm not sure if the the > OSD process gives the recovery threads a different disk priority or if > changing the scheduler without restarting the OSD allows the OSD to > use disk priorities. > * Reduced the number of osd_max_backfills from 2 to 1. > * Tried setting noin to give the new OSDs time to get the PG map and > peer before starting the backfill. This caused more problems than > solved as we had blocked I/O (over 200 seconds) until we set the new > OSDs to in.
You can also try to lower this settings (from the default): "osd_backfill_scan_min": "64", "osd_backfill_scan_max": "512", In our case we've set them to 1 and 8. And it helps a lot but recovery will take more time. -- PS _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
