Dear all,
I am reaching out regarding an issue with our Ceph cluster that has been
recurring every six hours. Upon investigating the problem using the "ceph
daemon dump_historic_slow_ops" command, I observed that the issue appears to be
related to slow operations, specifically getting stuck at "Waiting for RW
Locks." The wait times often range from one to two seconds.
Our cluster use SAS SSD disks from Samsung for the storage pool in question.
While these disks are of high quality and should provide sufficient speed, the
problem persists. The slow ops occurrence is consistent every six hours.
I would greatly appreciate any insights or suggestions you may have to address
and resolve this issue. If there are specific optimizations or configurations
that could improve the situation, please advise.
below are some output:
root@lasas003:~# ceph -v
ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus (stable)
"events": [
{
"event": "initiated",
"time": "2023-12-06T08:34:18.501644-0800",
"duration": 0
},
{
"event": "throttled",
"time": "2023-12-06T08:34:18.501644-0800",
"duration": 3.067e-06
},
{
"event": "header_read",
"time": "2023-12-06T08:34:18.501647-0800",
"duration": 3.5429999999999998e-06
},
{
"event": "all_read",
"time": "2023-12-06T08:34:18.501650-0800",
"duration": 9.3399999999999997e-07
},
{
"event": "dispatched",
"time": "2023-12-06T08:34:18.501651-0800",
"duration": 3.2830000000000002e-06
},
{
"event": "queued_for_pg",
"time": "2023-12-06T08:34:18.501654-0800",
"duration": 1.3819939990000001
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:19.883648-0800",
"duration": 5.7980000000000002e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:19.883654-0800",
"duration": 4.2484711649999998
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:24.132125-0800",
"duration": 1.0667e-05
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:24.132136-0800",
"duration": 2.1593527840000002
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:26.291489-0800",
"duration": 3.292e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:26.291492-0800",
"duration": 0.43918164700000001
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:26.730674-0800",
"duration": 5.1529999999999996e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:26.730679-0800",
"duration": 1.0531516869999999
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:27.783831-0800",
"duration": 5.1329999999999998e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:27.783836-0800",
"duration": 1.232525088
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:29.016361-0800",
"duration": 3.844e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:29.016365-0800",
"duration": 0.0051385700000000003
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:29.021503-0800",
"duration": 4.7600000000000002e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:29.021508-0800",
"duration": 0.0092808779999999994
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:29.030789-0800",
"duration": 4.0690000000000003e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:29.030793-0800",
"duration": 0.55757725499999999
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:29.588370-0800",
"duration": 5.5060000000000003e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:29.588376-0800",
"duration": 0.0064168929999999999
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:29.594793-0800",
"duration": 7.0690000000000004e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:29.594800-0800",
"duration": 0.0026404089999999998
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:29.597440-0800",
"duration": 3.3440000000000001e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:29.597444-0800",
"duration": 0.0051126670000000004
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:29.602556-0800",
"duration": 5.0200000000000002e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:29.602561-0800",
"duration": 0.0040569960000000002
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:29.606618-0800",
"duration": 5.0989999999999998e-06
},
{
"event": "waiting for rw locks",
"time": "2023-12-06T08:34:29.606623-0800",
"duration": 0.0068874100000000001
},
{
"event": "reached_pg",
"time": "2023-12-06T08:34:29.613511-0800",
"duration": 1.4636e-05
},
{
"event": "started",
"time": "2023-12-06T08:34:29.613525-0800",
"duration": 0.00028943699999999997
},
{
"event": "done",
"time": "2023-12-06T08:34:29.613815-0800",
"duration": 11.112171102
}
Thank you in advance for your assistance.
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
