Re: [ceph-users] Debugging 'slow requests' ...

2019-02-11 Thread Brad Hubbard
Glad to help! On Tue, Feb 12, 2019 at 4:55 PM Massimo Sgaravatto wrote: > > Thanks a lot Brad ! > > The problem is indeed in the network: we moved the OSD nodes back to the > "old" switches and the problem disappeared. > > Now we have to figure out what is wrong/misconfigured with the new switch

Re: [ceph-users] Debugging 'slow requests' ...

2019-02-11 Thread Massimo Sgaravatto
Thanks a lot Brad ! The problem is indeed in the network: we moved the OSD nodes back to the "old" switches and the problem disappeared. Now we have to figure out what is wrong/misconfigured with the new switch: we would try to replicate the problem, possibly without a ceph deployment ... Thanks

Re: [ceph-users] Debugging 'slow requests' ...

2019-02-09 Thread Brad Hubbard
The log ends at $ zcat ceph-osd.5.log.gz |tail -2 2019-02-09 07:37:00.022534 7f5fce60d700 1 -- 192.168.61.202:6816/157436 >> - conn(0x56308edcf000 :6816 s=STATE_ACCEPTING pgs=0 cs=0 l=0)._process_connection sd=296 - The last two messages are outbound to 192.168.222.204 and there are no further m

Re: [ceph-users] Debugging 'slow requests' ...

2019-02-09 Thread Massimo Sgaravatto
Thanks for your feedback ! I increased debug_ms to 1/5. This is another slow request (full output from 'ceph daemon osd.5 dump_historic_ops' for this event is attached): { "description": "osd_op(client.171725953.0:404377591 8.9b 8:d90adab6: ::rbd_data.c47f3c390c8495.

Re: [ceph-users] Debugging 'slow requests' ...

2019-02-08 Thread Brad Hubbard
Try capturing another log with debug_ms turned up. 1 or 5 should be Ok to start with. On Fri, Feb 8, 2019 at 8:37 PM Massimo Sgaravatto wrote: > > Our Luminous ceph cluster have been worked without problems for a while, but > in the last days we have been suffering from continuous slow requests.