A repop is a sub-operation between primaries and replicas mostly. That op only shows a duration of 1.3 seconds and the delay you mentioned previously was under a second. Do you see larger delays? Are they always between "sub_op_committed" and "commit_sent"?
What is your workload and how heavily utilised is your cluster/network? How hard are the underlying disks working? On Thu, Mar 21, 2019 at 4:11 PM Glen Baars <[email protected]> wrote: > > Hello Brad, > > It doesn't seem to be a set of OSDs, the cluster has 160ish OSDs over 9 hosts. > > I seem to get a lot of these ops also that don't show a client. > > "description": "osd_repop(client.14349712.0:4866968 15.36 > e30675/22264 15:6dd17247:::rbd_data.2359ef6b8b4567.000000000042766 > a:head v 30675'5522366)", > "initiated_at": "2019-03-21 16:51:56.862447", > "age": 376.527241, > "duration": 1.331278, > > Kind regards, > Glen Baars > > -----Original Message----- > From: Brad Hubbard <[email protected]> > Sent: Thursday, 21 March 2019 1:43 PM > To: Glen Baars <[email protected]> > Cc: [email protected] > Subject: Re: [ceph-users] Slow OPS > > Actually, the lag is between "sub_op_committed" and "commit_sent". Is there > any pattern to these slow requests? Do they involve the same osd, or set of > osds? > > On Thu, Mar 21, 2019 at 3:37 PM Brad Hubbard <[email protected]> wrote: > > > > On Thu, Mar 21, 2019 at 3:20 PM Glen Baars <[email protected]> > > wrote: > > > > > > Thanks for that - we seem to be experiencing the wait in this section of > > > the ops. > > > > > > { > > > "time": "2019-03-21 14:12:42.830191", > > > "event": "sub_op_committed" > > > }, > > > { > > > "time": "2019-03-21 14:12:43.699872", > > > "event": "commit_sent" > > > }, > > > > > > Does anyone know what that section is waiting for? > > > > Hi Glen, > > > > These are documented, to some extent, here. > > > > http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting > > -osd/ > > > > It looks like it may be taking a long time to communicate the commit > > message back to the client? Are these slow ops always the same client? > > > > > > > > Kind regards, > > > Glen Baars > > > > > > -----Original Message----- > > > From: Brad Hubbard <[email protected]> > > > Sent: Thursday, 21 March 2019 8:23 AM > > > To: Glen Baars <[email protected]> > > > Cc: [email protected] > > > Subject: Re: [ceph-users] Slow OPS > > > > > > On Thu, Mar 21, 2019 at 12:11 AM Glen Baars <[email protected]> > > > wrote: > > > > > > > > Hello Ceph Users, > > > > > > > > > > > > > > > > Does anyone know what the flag point ‘Started’ is? Is that ceph osd > > > > daemon waiting on the disk subsystem? > > > > > > This is set by "mark_started()" and is roughly set when the pg starts > > > processing the op. Might want to capture dump_historic_ops output after > > > the op completes. > > > > > > > > > > > > > > > > > > > Ceph 13.2.4 on centos 7.5 > > > > > > > > > > > > > > > > "description": "osd_op(client.1411875.0:422573570 > > > > 5.18ds0 > > > > 5:b1ed18e5:::rbd_data.6.cf7f46b8b4567.000000000046e41a:head [read > > > > > > > > 1703936~16384] snapc 0=[] ondisk+read+known_if_redirected > > > > e30622)", > > > > > > > > "initiated_at": "2019-03-21 01:04:40.598438", > > > > > > > > "age": 11.340626, > > > > > > > > "duration": 11.342846, > > > > > > > > "type_data": { > > > > > > > > "flag_point": "started", > > > > > > > > "client_info": { > > > > > > > > "client": "client.1411875", > > > > > > > > "client_addr": "10.4.37.45:0/627562602", > > > > > > > > "tid": 422573570 > > > > > > > > }, > > > > > > > > "events": [ > > > > > > > > { > > > > > > > > "time": "2019-03-21 01:04:40.598438", > > > > > > > > "event": "initiated" > > > > > > > > }, > > > > > > > > { > > > > > > > > "time": "2019-03-21 01:04:40.598438", > > > > > > > > "event": "header_read" > > > > > > > > }, > > > > > > > > { > > > > > > > > "time": "2019-03-21 01:04:40.598439", > > > > > > > > "event": "throttled" > > > > > > > > }, > > > > > > > > { > > > > > > > > "time": "2019-03-21 01:04:40.598450", > > > > > > > > "event": "all_read" > > > > > > > > }, > > > > > > > > { > > > > > > > > "time": "2019-03-21 01:04:40.598499", > > > > > > > > "event": "dispatched" > > > > > > > > }, > > > > > > > > { > > > > > > > > "time": "2019-03-21 01:04:40.598504", > > > > > > > > "event": "queued_for_pg" > > > > > > > > }, > > > > > > > > { > > > > > > > > "time": "2019-03-21 01:04:40.598883", > > > > > > > > "event": "reached_pg" > > > > > > > > }, > > > > > > > > { > > > > > > > > "time": "2019-03-21 01:04:40.598905", > > > > > > > > "event": "started" > > > > > > > > } > > > > > > > > ] > > > > > > > > } > > > > > > > > } > > > > > > > > ], > > > > > > > > > > > > > > > > Glen > > > > > > > > This e-mail is intended solely for the benefit of the addressee(s) and > > > > any other named recipient. It is confidential and may contain legally > > > > privileged or confidential information. If you are not the recipient, > > > > any use, distribution, disclosure or copying of this e-mail is > > > > prohibited. The confidentiality and legal privilege attached to this > > > > communication is not waived or lost by reason of the mistaken > > > > transmission or delivery to you. If you have received this e-mail in > > > > error, please notify us immediately. > > > > _______________________________________________ > > > > ceph-users mailing list > > > > [email protected] > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > -- > > > Cheers, > > > Brad > > > This e-mail is intended solely for the benefit of the addressee(s) and > > > any other named recipient. It is confidential and may contain legally > > > privileged or confidential information. If you are not the recipient, any > > > use, distribution, disclosure or copying of this e-mail is prohibited. > > > The confidentiality and legal privilege attached to this communication is > > > not waived or lost by reason of the mistaken transmission or delivery to > > > you. If you have received this e-mail in error, please notify us > > > immediately. > > > > > > > > -- > > Cheers, > > Brad > > > > -- > Cheers, > Brad > This e-mail is intended solely for the benefit of the addressee(s) and any > other named recipient. It is confidential and may contain legally privileged > or confidential information. If you are not the recipient, any use, > distribution, disclosure or copying of this e-mail is prohibited. The > confidentiality and legal privilege attached to this communication is not > waived or lost by reason of the mistaken transmission or delivery to you. If > you have received this e-mail in error, please notify us immediately. -- Cheers, Brad _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
