Hello Brad,
It doesn't seem to be a set of OSDs, the cluster has 160ish OSDs over 9 hosts.
I seem to get a lot of these ops also that don't show a client.
"description": "osd_repop(client.14349712.0:4866968 15.36
e30675/22264 15:6dd17247:::rbd_data.2359ef6b8b4567.000000000042766
a:head v 30675'5522366)",
"initiated_at": "2019-03-21 16:51:56.862447",
"age": 376.527241,
"duration": 1.331278,
Kind regards,
Glen Baars
-----Original Message-----
From: Brad Hubbard <[email protected]>
Sent: Thursday, 21 March 2019 1:43 PM
To: Glen Baars <[email protected]>
Cc: [email protected]
Subject: Re: [ceph-users] Slow OPS
Actually, the lag is between "sub_op_committed" and "commit_sent". Is there any
pattern to these slow requests? Do they involve the same osd, or set of osds?
On Thu, Mar 21, 2019 at 3:37 PM Brad Hubbard <[email protected]> wrote:
>
> On Thu, Mar 21, 2019 at 3:20 PM Glen Baars <[email protected]>
> wrote:
> >
> > Thanks for that - we seem to be experiencing the wait in this section of
> > the ops.
> >
> > {
> > "time": "2019-03-21 14:12:42.830191",
> > "event": "sub_op_committed"
> > },
> > {
> > "time": "2019-03-21 14:12:43.699872",
> > "event": "commit_sent"
> > },
> >
> > Does anyone know what that section is waiting for?
>
> Hi Glen,
>
> These are documented, to some extent, here.
>
> http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting
> -osd/
>
> It looks like it may be taking a long time to communicate the commit
> message back to the client? Are these slow ops always the same client?
>
> >
> > Kind regards,
> > Glen Baars
> >
> > -----Original Message-----
> > From: Brad Hubbard <[email protected]>
> > Sent: Thursday, 21 March 2019 8:23 AM
> > To: Glen Baars <[email protected]>
> > Cc: [email protected]
> > Subject: Re: [ceph-users] Slow OPS
> >
> > On Thu, Mar 21, 2019 at 12:11 AM Glen Baars <[email protected]>
> > wrote:
> > >
> > > Hello Ceph Users,
> > >
> > >
> > >
> > > Does anyone know what the flag point ‘Started’ is? Is that ceph osd
> > > daemon waiting on the disk subsystem?
> >
> > This is set by "mark_started()" and is roughly set when the pg starts
> > processing the op. Might want to capture dump_historic_ops output after the
> > op completes.
> >
> > >
> > >
> > >
> > > Ceph 13.2.4 on centos 7.5
> > >
> > >
> > >
> > > "description": "osd_op(client.1411875.0:422573570
> > > 5.18ds0
> > > 5:b1ed18e5:::rbd_data.6.cf7f46b8b4567.000000000046e41a:head [read
> > >
> > > 1703936~16384] snapc 0=[] ondisk+read+known_if_redirected
> > > e30622)",
> > >
> > > "initiated_at": "2019-03-21 01:04:40.598438",
> > >
> > > "age": 11.340626,
> > >
> > > "duration": 11.342846,
> > >
> > > "type_data": {
> > >
> > > "flag_point": "started",
> > >
> > > "client_info": {
> > >
> > > "client": "client.1411875",
> > >
> > > "client_addr": "10.4.37.45:0/627562602",
> > >
> > > "tid": 422573570
> > >
> > > },
> > >
> > > "events": [
> > >
> > > {
> > >
> > > "time": "2019-03-21 01:04:40.598438",
> > >
> > > "event": "initiated"
> > >
> > > },
> > >
> > > {
> > >
> > > "time": "2019-03-21 01:04:40.598438",
> > >
> > > "event": "header_read"
> > >
> > > },
> > >
> > > {
> > >
> > > "time": "2019-03-21 01:04:40.598439",
> > >
> > > "event": "throttled"
> > >
> > > },
> > >
> > > {
> > >
> > > "time": "2019-03-21 01:04:40.598450",
> > >
> > > "event": "all_read"
> > >
> > > },
> > >
> > > {
> > >
> > > "time": "2019-03-21 01:04:40.598499",
> > >
> > > "event": "dispatched"
> > >
> > > },
> > >
> > > {
> > >
> > > "time": "2019-03-21 01:04:40.598504",
> > >
> > > "event": "queued_for_pg"
> > >
> > > },
> > >
> > > {
> > >
> > > "time": "2019-03-21 01:04:40.598883",
> > >
> > > "event": "reached_pg"
> > >
> > > },
> > >
> > > {
> > >
> > > "time": "2019-03-21 01:04:40.598905",
> > >
> > > "event": "started"
> > >
> > > }
> > >
> > > ]
> > >
> > > }
> > >
> > > }
> > >
> > > ],
> > >
> > >
> > >
> > > Glen
> > >
> > > This e-mail is intended solely for the benefit of the addressee(s) and
> > > any other named recipient. It is confidential and may contain legally
> > > privileged or confidential information. If you are not the recipient, any
> > > use, distribution, disclosure or copying of this e-mail is prohibited.
> > > The confidentiality and legal privilege attached to this communication is
> > > not waived or lost by reason of the mistaken transmission or delivery to
> > > you. If you have received this e-mail in error, please notify us
> > > immediately.
> > > _______________________________________________
> > > ceph-users mailing list
> > > [email protected]
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >
> > --
> > Cheers,
> > Brad
> > This e-mail is intended solely for the benefit of the addressee(s) and any
> > other named recipient. It is confidential and may contain legally
> > privileged or confidential information. If you are not the recipient, any
> > use, distribution, disclosure or copying of this e-mail is prohibited. The
> > confidentiality and legal privilege attached to this communication is not
> > waived or lost by reason of the mistaken transmission or delivery to you.
> > If you have received this e-mail in error, please notify us immediately.
>
>
>
> --
> Cheers,
> Brad
--
Cheers,
Brad
This e-mail is intended solely for the benefit of the addressee(s) and any
other named recipient. It is confidential and may contain legally privileged or
confidential information. If you are not the recipient, any use, distribution,
disclosure or copying of this e-mail is prohibited. The confidentiality and
legal privilege attached to this communication is not waived or lost by reason
of the mistaken transmission or delivery to you. If you have received this
e-mail in error, please notify us immediately.
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
