Hi Sean,
Cut some part of slow request log, please find the attachment.
Thinking about set scrub interval more longer(once a month maybe).
An irrelevant question, do you think deep scrub will bring the performance
down？
Best wishes,
Mika
2015-11-24 18:24 GMT+08:00 Sean Redmond <[email protected]>:
> Hi,
>
> That seems very odd - what do the logs say for the osds with slow requests?
>
> Thanks
>
> On Tue, Nov 24, 2015 at 2:20 AM, Mika c <[email protected]> wrote:
>
>> Hi Sean,
>> Yes, the cluster scrubbing status(scrub + deep scrub) is almost two
>> weeks.
>> And the result of execute `ceph pg dump | grep scrub` is empty.
>> But command of "ceph health" show there is "*16 pgs
>> active+clean+scrubbing+deep, 2** pgs active+clean+scrubbing*".
>> I have 2 osds have slow requests warning.
>> Is it releated？
>>
>>
>>
>> Best wishes,
>> Mika
>>
>>
>> 2015-11-23 17:59 GMT+08:00 Sean Redmond <[email protected]>:
>>
>>> Hi Mika,
>>>
>>> Have the scubs been running for a long time? Can you see what pool they
>>> are running on? You can check using `ceph pg dump | grep scrub`
>>>
>>> Thanks
>>>
>>> On Mon, Nov 23, 2015 at 9:32 AM, Mika c <[email protected]> wrote:
>>>
>>>> Hi cephers,
>>>> We are facing a scrub issue. Our CEPH cluster is using Trusty /
>>>> Hammer 0.94.1 and have almost 320 OSD disks on 10 nodes.
>>>> And there are more than 30,000 PGs on cluster.
>>>> The cluster works fine until last week. We found the cluster health
>>>> status start display "active+clean+scrubbing+deep".
>>>> Some PGs scrub ok but next second another PG start to scrubbing (or
>>>> deep scrubbing) everyday.
>>>> We did not change the parameters of scrub. It should scrub once per
>>>> day and deep scrub once per week.
>>>> Has anyone experiences with this incident？
>>>>
>>>>
>>>>
>>>> Best wishes,
>>>> Mika
>>>>
>>>>
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> [email protected]
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>
>>>>
>>>
>>
>
2015-11-25 03:39:23.101368 osd.238 192.168.11.3:6882/2469348 1334 : cluster
[WRN] slow request 30.316965 seconds old, received at 2015-11-25
03:38:52.784331: osd_op(client.7203403.0:16611642
pre.7202536.151695__shadow_.3qppd3o1hCO7fiuQJD18AOvttvcb5S7_3 [writefull 0~0]
36.c65d1f05 ack+ondisk+write+known_if_redirected e95048) currently waiting for
subops from 38,355
2015-11-25 03:39:23.101372 osd.238 192.168.11.3:6882/2469348 1335 : cluster
[WRN] slow request 30.313927 seconds old, received at 2015-11-25
03:38:52.787369: osd_op(client.7203403.0:16611643
pre.7202536.151695__shadow_.3qppd3o1hCO7fiuQJD18AOvttvcb5S7_3 [writefull
0~524288] 36.c65d1f05 ack+ondisk+write+known_if_redirected e95048) currently
waiting for subops from 38,355
2015-11-25 03:39:23.101375 osd.238 192.168.11.3:6882/2469348 1336 : cluster
[WRN] slow request 30.311336 seconds old, received at 2015-11-25
03:38:52.789960: osd_op(client.7203403.0:16611644
pre.7202536.151695__shadow_.3qppd3o1hCO7fiuQJD18AOvttvcb5S7_3 [write
524288~524288] 36.c65d1f05 ack+ondisk+write+known_if_redirected e95048)
currently waiting for subops from 38,355
2015-11-25 03:39:23.101379 osd.238 192.168.11.3:6882/2469348 1337 : cluster
[WRN] slow request 30.307592 seconds old, received at 2015-11-25
03:38:52.793703: osd_op(client.7203403.0:16611645
pre.7202536.151695__shadow_.3qppd3o1hCO7fiuQJD18AOvttvcb5S7_3 [write
1048576~524288] 36.c65d1f05 ack+ondisk+write+known_if_redirected e95048)
currently waiting for subops from 38,355
2015-11-25 03:40:46.289760 osd.384 192.168.12.2:6806/573231 1869 : cluster
[WRN] 3 slow requests, 3 included below; oldest blocked for > 30.988835 secs
2015-11-25 03:40:46.289765 osd.384 192.168.12.2:6806/573231 1870 : cluster
[WRN] slow request 30.988835 seconds old, received at 2015-11-25
03:40:15.300878: osd_op(client.8634664.0:13286759
pre.7202536.151695__shadow_.gqjbnZqLSwkjCfw0wI-I7-OO9fX_iPT_3 [writefull
0~524288] 36.371c0419 ack+ondisk+write+known_if_redirected e95048) currently
waiting for subops from 33,315
2015-11-25 03:40:46.289769 osd.384 192.168.12.2:6806/573231 1871 : cluster
[WRN] slow request 30.985285 seconds old, received at 2015-11-25
03:40:15.304428: osd_op(client.8634664.0:13286760
pre.7202536.151695__shadow_.gqjbnZqLSwkjCfw0wI-I7-OO9fX_iPT_3 [write
524288~524288] 36.371c0419 ack+ondisk+write+known_if_redirected e95048)
currently waiting for subops from 33,315
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
