Re: [ceph-users] cephfs performance issue MDSs report slow requests and osd memory usage

[Robert LeBlanc]

On Tue, Sep 24, 2019 at 4:33 AM Thomas <74cmo...@gmail.com> wrote:
>
> Hi,
>
> I'm experiencing the same issue with this setting in ceph.conf:
> osd op queue = wpq
> osd op queue cut off = high
>
> Furthermore I cannot read any old data in the relevant pool that is
> serving CephFS.
> However, I can write new data and read this new data.

If you restarted all the OSDs with this setting, it won't necessarily
prevent any blocked IO, it just really helps prevent the really long
blocked IO and makes sure that IO is eventually done in a more fair
manner.

It sounds like you may have some MDS issues that are deeper than my
understanding. First thing I'd try is to bounce the MDS service.

> > If I want to add this my ceph-ansible playbook parameters, in which files I 
> > should add it and what is the best way to do it ?
> >
> > Add those 3 lines in all.yml or osds.yml ?
> >
> > ceph_conf_overrides:
> >   global:
> > osd_op_queue_cut_off: high
> >
> > Is there another (better?) way to do that?

I can't speak to either of those approaches. I wanted all my config in
a single file, so I put it in my inventory file, but it looks like you
have the right idea.


Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs performance issue MDSs report slow requests and osd memory usage

[Thomas]

Hi,

I'm experiencing the same issue with this setting in ceph.conf:
    osd op queue = wpq
    osd op queue cut off = high

Furthermore I cannot read any old data in the relevant pool that is
serving CephFS.
However, I can write new data and read this new data.

Regards
Thomas

Am 24.09.2019 um 10:24 schrieb Yoann Moulin:
> Hello,
>
>>> I have a Ceph Nautilus Cluster 14.2.1 for cephfs only on 40x 1.8T SAS disk 
>>> (no SSD) in 20 servers.
>>>
>>> I often get "MDSs report slow requests" and plenty of "[WRN] 3 slow 
>>> requests, 0 included below; oldest blocked for > 60281.199503 secs"
>>>
>>> After a few investigations, I saw that ALL ceph-osd process eat a lot of 
>>> memory, up to 130GB RSS each. It this value normal? May this related to
>>> slow requests? Is disk only increasing the probability to get slow requests?
>> If you haven't set:
>>
>> osd op queue cut off = high
>>
>> in /etc/ceph/ceph.conf on your OSDs, I'd give that a try. It should
>> help quite a bit with pure HDD clusters.
> OK I'll try this, thanks.
>
> If I want to add this my ceph-ansible playbook parameters, in which files I 
> should add it and what is the best way to do it ?
>
> Add those 3 lines in all.yml or osds.yml ?
>
> ceph_conf_overrides:
>   global:
> osd_op_queue_cut_off: high
>
> Is there another (better?) way to do that?
>
> Thanks for your help.
>
> Best regards,
>

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs performance issue MDSs report slow requests and osd memory usage

[Yoann Moulin]

Hello,

>> I have a Ceph Nautilus Cluster 14.2.1 for cephfs only on 40x 1.8T SAS disk 
>> (no SSD) in 20 servers.
>>
>> I often get "MDSs report slow requests" and plenty of "[WRN] 3 slow 
>> requests, 0 included below; oldest blocked for > 60281.199503 secs"
>>
>> After a few investigations, I saw that ALL ceph-osd process eat a lot of 
>> memory, up to 130GB RSS each. It this value normal? May this related to
>> slow requests? Is disk only increasing the probability to get slow requests?
>
> If you haven't set:
> 
> osd op queue cut off = high
> 
> in /etc/ceph/ceph.conf on your OSDs, I'd give that a try. It should
> help quite a bit with pure HDD clusters.

OK I'll try this, thanks.

If I want to add this my ceph-ansible playbook parameters, in which files I 
should add it and what is the best way to do it ?

Add those 3 lines in all.yml or osds.yml ?

ceph_conf_overrides:
  global:
osd_op_queue_cut_off: high

Is there another (better?) way to do that?

Thanks for your help.

Best regards,

-- 
Yoann Moulin
EPFL IC-IT
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs performance issue MDSs report slow requests and osd memory usage

[Robert LeBlanc]

On Thu, Sep 19, 2019 at 2:36 AM Yoann Moulin  wrote:
>
> Hello,
>
> I have a Ceph Nautilus Cluster 14.2.1 for cephfs only on 40x 1.8T SAS disk 
> (no SSD) in 20 servers.
>
> >   cluster:
> > id: 778234df-5784-4021-b983-0ee1814891be
> > health: HEALTH_WARN
> > 2 MDSs report slow requests
> >
> >   services:
> > mon: 3 daemons, quorum icadmin006,icadmin007,icadmin008 (age 5d)
> > mgr: icadmin008(active, since 18h), standbys: icadmin007, icadmin006
> > mds: cephfs:3 
> > {0=icadmin006=up:active,1=icadmin007=up:active,2=icadmin008=up:active}
> > osd: 40 osds: 40 up (since 2w), 40 in (since 3w)
> >
> >   data:
> > pools:   3 pools, 672 pgs
> > objects: 36.08M objects, 19 TiB
> > usage:   51 TiB used, 15 TiB / 65 TiB avail
> > pgs: 670 active+clean
> >  2   active+clean+scrubbing
>
> I often get "MDSs report slow requests" and plenty of "[WRN] 3 slow requests, 
> 0 included below; oldest blocked for > 60281.199503 secs"
>
> > HEALTH_WARN 2 MDSs report slow requests
> > MDS_SLOW_REQUEST 2 MDSs report slow requests
> > mdsicadmin007(mds.1): 3 slow requests are blocked > 30 secs
> > mdsicadmin006(mds.0): 10 slow requests are blocked > 30 secs
>
> After a few investigations, I saw that ALL ceph-osd process eat a lot of 
> memory, up to 130GB RSS each. It this value normal? May this related to
> slow requests? Is disk only increasing the probability to get slow requests?
>
> > USER PID %CPU %MEM   VSZ   RSS TTY STAT STAR   TIME COMMAND
> > ceph   34196  3.6 35.0 156247524 138521572 ? Ssl  Jul01 4173:18 
> > /usr/bin/ceph-osd -f --cluster apollo --id 1 --setuser ceph --setgroup ceph
> > ceph   34394  3.6 35.0 160001436 138487776 ? Ssl  Jul01 4178:37 
> > /usr/bin/ceph-osd -f --cluster apollo --id 32 --setuser ceph --setgroup ceph
> > ceph   34709  3.5 35.1 156369636 138752044 ? Ssl  Jul01 4088:57 
> > /usr/bin/ceph-osd -f --cluster apollo --id 29 --setuser ceph --setgroup ceph
> > ceph   34915  3.4 35.1 158976936 138715900 ? Ssl  Jul01 3950:45 
> > /usr/bin/ceph-osd -f --cluster apollo --id 3 --setuser ceph --setgroup ceph
> > ceph   34156  3.4 35.1 158280768 138714484 ? Ssl  Jul01 3984:11 
> > /usr/bin/ceph-osd -f --cluster apollo --id 30 --setuser ceph --setgroup ceph
> > ceph   34378  3.7 35.1 155162420 138708096 ? Ssl  Jul01 4312:12 
> > /usr/bin/ceph-osd -f --cluster apollo --id 8 --setuser ceph --setgroup ceph
> > ceph   34161  3.5 35.0 159606788 138523652 ? Ssl  Jul01 4128:17 
> > /usr/bin/ceph-osd -f --cluster apollo --id 16 --setuser ceph --setgroup ceph
> > ceph   34380  3.6 35.1 161465372 138670168 ? Ssl  Jul01 4238:20 
> > /usr/bin/ceph-osd -f --cluster apollo --id 35 --setuser ceph --setgroup ceph
> > ceph   33822  3.7 35.1 163456644 138734036 ? Ssl  Jul01 4342:05 
> > /usr/bin/ceph-osd -f --cluster apollo --id 15 --setuser ceph --setgroup ceph
> > ceph   34003  3.8 35.0 161868584 138531208 ? Ssl  Jul01 4427:32 
> > /usr/bin/ceph-osd -f --cluster apollo --id 38 --setuser ceph --setgroup ceph
> > ceph9753  2.8 24.2 96923856 95580776 ?   Ssl  Sep02 700:25 
> > /usr/bin/ceph-osd -f --cluster apollo --id 31 --setuser ceph --setgroup ceph
> > ceph   10120  2.5 24.0 96130340 94856244 ?   Ssl  Sep02 644:50 
> > /usr/bin/ceph-osd -f --cluster apollo --id 7 --setuser ceph --setgroup ceph
> > ceph   36204  3.6 35.0 159394476 138592124 ? Ssl  Jul01 4185:36 
> > /usr/bin/ceph-osd -f --cluster apollo --id 18 --setuser ceph --setgroup ceph
> > ceph   36427  3.7 34.4 155699060 136076432 ? Ssl  Jul01 4298:26 
> > /usr/bin/ceph-osd -f --cluster apollo --id 36 --setuser ceph --setgroup ceph
> > ceph   36622  4.1 35.1 158219408 138724688 ? Ssl  Jul01 4779:14 
> > /usr/bin/ceph-osd -f --cluster apollo --id 19 --setuser ceph --setgroup ceph
> > ceph   36881  4.0 35.1 157748752 138719064 ? Ssl  Jul01 4669:54 
> > /usr/bin/ceph-osd -f --cluster apollo --id 37 --setuser ceph --setgroup ceph
> > ceph   34649  3.7 35.1 159601580 138652012 ? Ssl  Jul01 4337:20 
> > /usr/bin/ceph-osd -f --cluster apollo --id 14 --setuser ceph --setgroup ceph
> > ceph   34881  3.8 35.1 158632412 138764376 ? Ssl  Jul01 4433:50 
> > /usr/bin/ceph-osd -f --cluster apollo --id 33 --setuser ceph --setgroup ceph
> > ceph   34646  4.2 35.1 155029328 138732376 ? Ssl  Jul01 4831:24 
> > /usr/bin/ceph-osd -f --cluster apollo --id 17 --setuser ceph --setgroup ceph
> > ceph   34881  4.1 35.1 156801676 138763588 ? Ssl  Jul01 4710:19 
> > /usr/bin/ceph-osd -f --cluster apollo --id 39 --setuser ceph --setgroup ceph
> > ceph   36766  3.7 35.1 158070740 138703240 ? Ssl  Jul01 4341:42 
> > /usr/bin/ceph-osd -f --cluster apollo --id 13 --setuser ceph --setgroup ceph
> > ceph   37013  3.5 35.0 157767668 138272248 ? Ssl  Jul01 4094:12 
> > /usr/bin/ceph-osd -f --cluster apollo --id 34 --setuser ceph --setgroup ceph
> > ceph   35007  3.4 35.1 160318780 138756404 ? Ssl  Jul01 3963:21 

[ceph-users] cephfs performance issue MDSs report slow requests and osd memory usage

[Yoann Moulin]

Hello,

I have a Ceph Nautilus Cluster 14.2.1 for cephfs only on 40x 1.8T SAS disk (no 
SSD) in 20 servers.

>   cluster:
> id: 778234df-5784-4021-b983-0ee1814891be
> health: HEALTH_WARN
> 2 MDSs report slow requests
>  
>   services:
> mon: 3 daemons, quorum icadmin006,icadmin007,icadmin008 (age 5d)
> mgr: icadmin008(active, since 18h), standbys: icadmin007, icadmin006
> mds: cephfs:3 
> {0=icadmin006=up:active,1=icadmin007=up:active,2=icadmin008=up:active}
> osd: 40 osds: 40 up (since 2w), 40 in (since 3w)
>  
>   data:
> pools:   3 pools, 672 pgs
> objects: 36.08M objects, 19 TiB
> usage:   51 TiB used, 15 TiB / 65 TiB avail
> pgs: 670 active+clean
>  2   active+clean+scrubbing

I often get "MDSs report slow requests" and plenty of "[WRN] 3 slow requests, 0 
included below; oldest blocked for > 60281.199503 secs"

> HEALTH_WARN 2 MDSs report slow requests
> MDS_SLOW_REQUEST 2 MDSs report slow requests
> mdsicadmin007(mds.1): 3 slow requests are blocked > 30 secs
> mdsicadmin006(mds.0): 10 slow requests are blocked > 30 secs

After a few investigations, I saw that ALL ceph-osd process eat a lot of 
memory, up to 130GB RSS each. It this value normal? May this related to
slow requests? Is disk only increasing the probability to get slow requests?

> USER PID %CPU %MEM   VSZ   RSS TTY STAT STAR   TIME COMMAND
> ceph   34196  3.6 35.0 156247524 138521572 ? Ssl  Jul01 4173:18 
> /usr/bin/ceph-osd -f --cluster apollo --id 1 --setuser ceph --setgroup ceph
> ceph   34394  3.6 35.0 160001436 138487776 ? Ssl  Jul01 4178:37 
> /usr/bin/ceph-osd -f --cluster apollo --id 32 --setuser ceph --setgroup ceph
> ceph   34709  3.5 35.1 156369636 138752044 ? Ssl  Jul01 4088:57 
> /usr/bin/ceph-osd -f --cluster apollo --id 29 --setuser ceph --setgroup ceph
> ceph   34915  3.4 35.1 158976936 138715900 ? Ssl  Jul01 3950:45 
> /usr/bin/ceph-osd -f --cluster apollo --id 3 --setuser ceph --setgroup ceph
> ceph   34156  3.4 35.1 158280768 138714484 ? Ssl  Jul01 3984:11 
> /usr/bin/ceph-osd -f --cluster apollo --id 30 --setuser ceph --setgroup ceph
> ceph   34378  3.7 35.1 155162420 138708096 ? Ssl  Jul01 4312:12 
> /usr/bin/ceph-osd -f --cluster apollo --id 8 --setuser ceph --setgroup ceph
> ceph   34161  3.5 35.0 159606788 138523652 ? Ssl  Jul01 4128:17 
> /usr/bin/ceph-osd -f --cluster apollo --id 16 --setuser ceph --setgroup ceph
> ceph   34380  3.6 35.1 161465372 138670168 ? Ssl  Jul01 4238:20 
> /usr/bin/ceph-osd -f --cluster apollo --id 35 --setuser ceph --setgroup ceph
> ceph   33822  3.7 35.1 163456644 138734036 ? Ssl  Jul01 4342:05 
> /usr/bin/ceph-osd -f --cluster apollo --id 15 --setuser ceph --setgroup ceph
> ceph   34003  3.8 35.0 161868584 138531208 ? Ssl  Jul01 4427:32 
> /usr/bin/ceph-osd -f --cluster apollo --id 38 --setuser ceph --setgroup ceph
> ceph9753  2.8 24.2 96923856 95580776 ?   Ssl  Sep02 700:25 
> /usr/bin/ceph-osd -f --cluster apollo --id 31 --setuser ceph --setgroup ceph
> ceph   10120  2.5 24.0 96130340 94856244 ?   Ssl  Sep02 644:50 
> /usr/bin/ceph-osd -f --cluster apollo --id 7 --setuser ceph --setgroup ceph
> ceph   36204  3.6 35.0 159394476 138592124 ? Ssl  Jul01 4185:36 
> /usr/bin/ceph-osd -f --cluster apollo --id 18 --setuser ceph --setgroup ceph
> ceph   36427  3.7 34.4 155699060 136076432 ? Ssl  Jul01 4298:26 
> /usr/bin/ceph-osd -f --cluster apollo --id 36 --setuser ceph --setgroup ceph
> ceph   36622  4.1 35.1 158219408 138724688 ? Ssl  Jul01 4779:14 
> /usr/bin/ceph-osd -f --cluster apollo --id 19 --setuser ceph --setgroup ceph
> ceph   36881  4.0 35.1 157748752 138719064 ? Ssl  Jul01 4669:54 
> /usr/bin/ceph-osd -f --cluster apollo --id 37 --setuser ceph --setgroup ceph
> ceph   34649  3.7 35.1 159601580 138652012 ? Ssl  Jul01 4337:20 
> /usr/bin/ceph-osd -f --cluster apollo --id 14 --setuser ceph --setgroup ceph
> ceph   34881  3.8 35.1 158632412 138764376 ? Ssl  Jul01 4433:50 
> /usr/bin/ceph-osd -f --cluster apollo --id 33 --setuser ceph --setgroup ceph
> ceph   34646  4.2 35.1 155029328 138732376 ? Ssl  Jul01 4831:24 
> /usr/bin/ceph-osd -f --cluster apollo --id 17 --setuser ceph --setgroup ceph
> ceph   34881  4.1 35.1 156801676 138763588 ? Ssl  Jul01 4710:19 
> /usr/bin/ceph-osd -f --cluster apollo --id 39 --setuser ceph --setgroup ceph
> ceph   36766  3.7 35.1 158070740 138703240 ? Ssl  Jul01 4341:42 
> /usr/bin/ceph-osd -f --cluster apollo --id 13 --setuser ceph --setgroup ceph
> ceph   37013  3.5 35.0 157767668 138272248 ? Ssl  Jul01 4094:12 
> /usr/bin/ceph-osd -f --cluster apollo --id 34 --setuser ceph --setgroup ceph
> ceph   35007  3.4 35.1 160318780 138756404 ? Ssl  Jul01 3963:21 
> /usr/bin/ceph-osd -f --cluster apollo --id 2 --setuser ceph --setgroup ceph
> ceph   35217  3.5 35.1 159023744 138626680 ? Ssl  Jul01 4041:50 
> /usr/bin/ceph-osd -f --cluster apollo --id 22 --setuser 

Re: [ceph-users] cephfs performance issue

[Ouyang Xu]

Hi David:

That's works, thank you very much!

Best regards,

Steven

On 2018年03月29日 18:30, David C wrote:

Pretty sure you're getting stung by: http://tracker.ceph.com/issues/17563

Consider using an elrepo kernel, 4.14 works well for me.



On Thu, 29 Mar 2018, 09:46 Dan van der Ster, > wrote:


On Thu, Mar 29, 2018 at 10:31 AM, Robert Sander
> wrote:
> On 29.03.2018 09:50, ouyangxu wrote:
>
>> I'm using Ceph 12.2.4 with CentOS 7.4, and tring to use cephfs for
>> MariaDB deployment,
>
> Don't do this.
> As the old saying goes: If it hurts, stop doing it.

Why not? Let's find out where and why the perf is lacking, then
fix it!

-- dan
___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs performance issue

[David C]

Pretty sure you're getting stung by: http://tracker.ceph.com/issues/17563

Consider using an elrepo kernel, 4.14 works well for me.



On Thu, 29 Mar 2018, 09:46 Dan van der Ster,  wrote:

> On Thu, Mar 29, 2018 at 10:31 AM, Robert Sander
>  wrote:
> > On 29.03.2018 09:50, ouyangxu wrote:
> >
> >> I'm using Ceph 12.2.4 with CentOS 7.4, and tring to use cephfs for
> >> MariaDB deployment,
> >
> > Don't do this.
> > As the old saying goes: If it hurts, stop doing it.
>
> Why not? Let's find out where and why the perf is lacking, then fix it!
>
> -- dan
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs performance issue

[Dan van der Ster]

On Thu, Mar 29, 2018 at 10:31 AM, Robert Sander
 wrote:
> On 29.03.2018 09:50, ouyangxu wrote:
>
>> I'm using Ceph 12.2.4 with CentOS 7.4, and tring to use cephfs for
>> MariaDB deployment,
>
> Don't do this.
> As the old saying goes: If it hurts, stop doing it.

Why not? Let's find out where and why the perf is lacking, then fix it!

-- dan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs performance issue

[Robert Sander]

On 29.03.2018 09:50, ouyangxu wrote:

> I'm using Ceph 12.2.4 with CentOS 7.4, and tring to use cephfs for
> MariaDB deployment,

Don't do this.
As the old saying goes: If it hurts, stop doing it.

Regards
-- 
Robert Sander
Heinlein Support GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] cephfs performance issue

[ouyangxu]

Hi Ceph users:

I'm using Ceph 12.2.4 with CentOS 7.4, and tring to use cephfs for MariaDB 
deployment, the configuration is default, but got very pool performance during 
creating tables, if I use the local file system, not this issue.

Here is the sql scripts I used:
[root@cmv01cn01]$ cat mysql_test.sql
CREATE TABLE test.t001 (col INT)\g
CREATE TABLE test.t002 (col INT)\g
CREATE TABLE test.t003 (col INT)\g
CREATE TABLE test.t004 (col INT)\g
CREATE TABLE test.t005 (col INT)\g
CREATE TABLE test.t006 (col INT)\g
CREATE TABLE test.t007 (col INT)\g
CREATE TABLE test.t008 (col INT)\g
CREATE TABLE test.t009 (col INT)\g
DROP TABLE test.t001\g
DROP TABLE test.t002\g
DROP TABLE test.t003\g
DROP TABLE test.t004\g
DROP TABLE test.t005\g
DROP TABLE test.t006\g
DROP TABLE test.t007\g
DROP TABLE test.t008\g
DROP TABLE test.t009\g

The following is the running result:
[root@cmv01cn01]$ mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 6522
Server version: 10.1.20-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> source mysql_test.sql
Query OK, 0 rows affected (3.26 sec)
Query OK, 0 rows affected (5.32 sec)
Query OK, 0 rows affected (4.53 sec)
Query OK, 0 rows affected (5.09 sec)
Query OK, 0 rows affected (4.96 sec)
Query OK, 0 rows affected (4.94 sec)
Query OK, 0 rows affected (4.96 sec)
Query OK, 0 rows affected (5.02 sec)
Query OK, 0 rows affected (5.08 sec)
Query OK, 0 rows affected (0.11 sec)
Query OK, 0 rows affected (0.07 sec)
Query OK, 0 rows affected (0.07 sec)
Query OK, 0 rows affected (0.10 sec)
Query OK, 0 rows affected (0.06 sec)
Query OK, 0 rows affected (0.10 sec)
Query OK, 0 rows affected (0.02 sec)
Query OK, 0 rows affected (0.02 sec)
Query OK, 0 rows affected (0.02 sec)
MariaDB [(none)]>

As you can see, the average time for creating table around 5s, regarding to 
drop table, it is acceptable.

I also dumped the ops for mds as below:

[root@cmv01sn01 ceph]# ceph daemon mds.cmv01sn01 dump_ops_in_flight
{
"ops": [
{
"description": "client_request(client.854369:4659 create 
#0x1003994/t2_0.frm 2018-03-28 17:24:53.744090 caller_uid=27, 
caller_gid=27{})",
"initiated_at": "2018-03-28 17:24:53.744827",
"age": 4.567939,
"duration": 4.567955,
"type_data": {
"flag_point": "submit entry: journal_and_reply",
"reqid": "client.854369:4659",
"op_type": "client_request",
"client_info": {
"client": "client.854369",
"tid": 4659
},
"events": [
{
"time": "2018-03-28 17:24:53.744827",
"event": "initiated"
},
{
"time": "2018-03-28 17:24:53.745226",
"event": "acquired locks"
},
{
"time": "2018-03-28 17:24:53.745364",
"event": "early_replied"
},
{
"time": "2018-03-28 17:24:53.745367",
"event": "submit entry: journal_and_reply"
}
]
}
},
{
"description": "client_request(client.854369:4660 create 
#0x1003994/t2_0.ibd 2018-03-28 17:24:53.751090 caller_uid=27, 
caller_gid=27{})",
"initiated_at": "2018-03-28 17:24:53.752039",
"age": 4.560727,
"duration": 4.560763,
"type_data": {
"flag_point": "submit entry: journal_and_reply",
"reqid": "client.854369:4660",
"op_type": "client_request",
"client_info": {
"client": "client.854369",
"tid": 4660
},
"events": [
{
"time": "2018-03-28 17:24:53.752039",
"event": "initiated"
},
{
"time": "2018-03-28 17:24:53.752358",
"event": "acquired locks"
},
{
"time": "2018-03-28 17:24:53.752480",
"event": "early_replied"
},
{
"time": "2018-03-28 17:24:53.752483",
"event": "submit entry: journal_and_reply"
}
]
}
}
],
"num_ops": 2
}

It seems like stuck at journal_and_reply.

So, does anyone faced this situation? Any thoughts are appropriated.

Thanks,

Steven
___
ceph-users mailing list