Re: [ceph-users] in retrospect get OSD for "slow requests are blocked" ? / get detailed health status via librados?
On Sat, May 19, 2018 at 5:01 PM, Uwe Sauter wrote: > The mistery is that these blocked requests occur numerously when at > least > one of the 6 servers is booted with kernel 4.15.17, if all are running > 4.13.16 the number of blocked requests is infrequent and low. Sounds like you need to profile your two kernel versions and work out why one is under-performing. >>> >>> Well, the problem is that I see this behavior only in our production >>> system (6 hosts and 22 OSDs total). The test system I have is >>> a bit smaller (only 3 hosts with 12 OSDs on older hardware) and shows no >>> sign of this possible regression… >> >> >> Are you saying you can't gather performance data from your production >> system? > > > As far as I can tell the issue only occurs on the production cluster. > Without a way to reproduce > on the test cluster I can't bisect the kernels as on the production cluster > runs our central > infrastructure and each time the active LDAP is stuck, most of the other > services are stuck as well… > My colleagues won't appreciate that. > > What other kind of performance data would you have collected? > On systems where this can be reproduced I would use tools like 'perf top', pvp, collectd and maybe something like the following to capture data that can be analysed to define the nature of the issue. // for rhel6 and rhel7 so may need modification # { top -n 5 -b > /tmp/top.out; \ vmstat 1 50 > /tmp/vm.out; \ iostat -tkx -p ALL 1 10 > /tmp/io.out; \ mpstat -A 1 10 > /tmp/mp.out; \ ps auwwx > /tmp/ps1.out; \ ps axHo %cpu,stat,pid,tid,pgid,ppid,comm,wchan > /tmp/ps2.out; \ sar -A 1 50 > /tmp/sar.out; \ free > /tmp/free.out; } ; tar -cjvf outputs_$(hostname)_$(date +"%d-%b-%Y_%H%M").tar.bz2 /tmp/*.out As you've already pointed out this currently seems to be a kernel performance issue but analysis of this sort of data should help you narrow it down. Of course, all of this relies on you being able to reproduce the issue, but maybe you can gather a baseline to begin with so you have something to compare to when you are in a position to gather perf data during an issue. At the same time I'd suggest pursuing this with Proxmox and/or Ubuntu to see if they have anything to offer. -- Cheers, Brad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] in retrospect get OSD for "slow requests are blocked" ? / get detailed health status via librados?
Am 19.05.2018 um 01:45 schrieb Brad Hubbard: On Thu, May 17, 2018 at 6:06 PM, Uwe Sauter wrote: Brad, thanks for the bug report. This is exactly the problem I am having (log-wise). You don't give any indication what version you are running but see https://tracker.ceph.com/issues/23205 the cluster is an Proxmox installation which is based on an Ubuntu kernel. # ceph -v ceph version 12.2.5 (dfcb7b53b2e4fcd2a5af0240d4975adc711ab96e) luminous (stable) The mistery is that these blocked requests occur numerously when at least one of the 6 servers is booted with kernel 4.15.17, if all are running 4.13.16 the number of blocked requests is infrequent and low. Sounds like you need to profile your two kernel versions and work out why one is under-performing. Well, the problem is that I see this behavior only in our production system (6 hosts and 22 OSDs total). The test system I have is a bit smaller (only 3 hosts with 12 OSDs on older hardware) and shows no sign of this possible regression… Are you saying you can't gather performance data from your production system? As far as I can tell the issue only occurs on the production cluster. Without a way to reproduce on the test cluster I can't bisect the kernels as on the production cluster runs our central infrastructure and each time the active LDAP is stuck, most of the other services are stuck as well… My colleagues won't appreciate that. What other kind of performance data would you have collected? Uwe Regards, Uwe ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] in retrospect get OSD for "slow requests are blocked" ? / get detailed health status via librados?
On Thu, May 17, 2018 at 6:06 PM, Uwe Sauter wrote: > Brad, > > thanks for the bug report. This is exactly the problem I am having (log-wise). You don't give any indication what version you are running but see https://tracker.ceph.com/issues/23205 >>> >>> >>> the cluster is an Proxmox installation which is based on an Ubuntu kernel. >>> >>> # ceph -v >>> ceph version 12.2.5 (dfcb7b53b2e4fcd2a5af0240d4975adc711ab96e) luminous >>> (stable) >>> >>> The mistery is that these blocked requests occur numerously when at least >>> one of the 6 servers is booted with kernel 4.15.17, if all are running >>> 4.13.16 the number of blocked requests is infrequent and low. >> >> Sounds like you need to profile your two kernel versions and work out >> why one is under-performing. >> > > Well, the problem is that I see this behavior only in our production system > (6 hosts and 22 OSDs total). The test system I have is > a bit smaller (only 3 hosts with 12 OSDs on older hardware) and shows no sign > of this possible regression… Are you saying you can't gather performance data from your production system? > > > Regards, > > Uwe -- Cheers, Brad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] in retrospect get OSD for "slow requests are blocked" ? / get detailed health status via librados?
Brad, thanks for the bug report. This is exactly the problem I am having (log-wise). >>> >>> You don't give any indication what version you are running but see >>> https://tracker.ceph.com/issues/23205 >> >> >> the cluster is an Proxmox installation which is based on an Ubuntu kernel. >> >> # ceph -v >> ceph version 12.2.5 (dfcb7b53b2e4fcd2a5af0240d4975adc711ab96e) luminous >> (stable) >> >> The mistery is that these blocked requests occur numerously when at least >> one of the 6 servers is booted with kernel 4.15.17, if all are running >> 4.13.16 the number of blocked requests is infrequent and low. > > Sounds like you need to profile your two kernel versions and work out > why one is under-performing. > Well, the problem is that I see this behavior only in our production system (6 hosts and 22 OSDs total). The test system I have is a bit smaller (only 3 hosts with 12 OSDs on older hardware) and shows no sign of this possible regression… Regards, Uwe ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] in retrospect get OSD for "slow requests are blocked" ? / get detailed health status via librados?
On Thu, May 17, 2018 at 4:16 PM, Uwe Sauter wrote: > Hi, > >>> I'm currently chewing on an issue regarding "slow requests are blocked". >>> I'd like to identify the OSD that is causing those events >>> once the cluster is back to HEALTH_OK (as I have no monitoring yet that >>> would get this info in realtime). >>> >>> Collecting this information could help identify aging disks if you were >>> able to accumulate and analyze which OSD had blocking >>> requests in the past and how often those events occur. >>> >>> My research so far let's me think that this information is only available >>> as long as the requests are actually blocked. Is this >>> correct? >> >> >> You don't give any indication what version you are running but see >> https://tracker.ceph.com/issues/23205 > > > the cluster is an Proxmox installation which is based on an Ubuntu kernel. > > # ceph -v > ceph version 12.2.5 (dfcb7b53b2e4fcd2a5af0240d4975adc711ab96e) luminous > (stable) > > The mistery is that these blocked requests occur numerously when at least > one of the 6 servers is booted with kernel 4.15.17, if all are running > 4.13.16 the number of blocked requests is infrequent and low. Sounds like you need to profile your two kernel versions and work out why one is under-performing. > > > Regards, > > Uwe -- Cheers, Brad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] in retrospect get OSD for "slow requests are blocked" ? / get detailed health status via librados?
Hi, I'm currently chewing on an issue regarding "slow requests are blocked". I'd like to identify the OSD that is causing those events once the cluster is back to HEALTH_OK (as I have no monitoring yet that would get this info in realtime). Collecting this information could help identify aging disks if you were able to accumulate and analyze which OSD had blocking requests in the past and how often those events occur. My research so far let's me think that this information is only available as long as the requests are actually blocked. Is this correct? You don't give any indication what version you are running but see https://tracker.ceph.com/issues/23205 the cluster is an Proxmox installation which is based on an Ubuntu kernel. # ceph -v ceph version 12.2.5 (dfcb7b53b2e4fcd2a5af0240d4975adc711ab96e) luminous (stable) The mistery is that these blocked requests occur numerously when at least one of the 6 servers is booted with kernel 4.15.17, if all are running 4.13.16 the number of blocked requests is infrequent and low. Regards, Uwe ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] in retrospect get OSD for "slow requests are blocked" ? / get detailed health status via librados?
On Wed, May 16, 2018 at 6:16 PM, Uwe Sauter wrote: > Hi folks, > > I'm currently chewing on an issue regarding "slow requests are blocked". I'd > like to identify the OSD that is causing those events > once the cluster is back to HEALTH_OK (as I have no monitoring yet that would > get this info in realtime). > > Collecting this information could help identify aging disks if you were able > to accumulate and analyze which OSD had blocking > requests in the past and how often those events occur. > > My research so far let's me think that this information is only available as > long as the requests are actually blocked. Is this > correct? You don't give any indication what version you are running but see https://tracker.ceph.com/issues/23205 > > MON logs only show that those events occure and how many requests are in > blocking state but no indication of which OSD is > affected. Is there a way to identify blocking requests from the OSD log files? > > > On a side note: I was trying to write a small Python script that would > extract this kind of information in realtime but while I > was able to register a MonitorLog callback that would receive the same > messages as you would get with "ceph -w" I haven's seen in > the librados Python bindings documentation the possibility to do the > equivalent of "ceph health detail". Any suggestions on how to > get the blocking OSDs via librados? > > > Thanks, > > Uwe > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Cheers, Brad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] in retrospect get OSD for "slow requests are blocked" ? / get detailed health status via librados?
On 05/16/2018 07:18 AM, Uwe Sauter wrote: > Hi Mohamad, > >> >> I think this is what you're looking for: >> >> $> ceph daemon osd.X dump_historic_slow_ops >> >> which gives you recent slow operations, as opposed to >> >> $> ceph daemon osd.X dump_blocked_ops >> >> which returns current blocked operations. You can also add a filter to >> those commands. > Thanks for these commands. I'll have a look into those. If I understand these > correctly it means that I need to run these at each > server for each OSD instead of at a central location, is that correct? > That's the case, as it uses the admin socket. Mohamad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] in retrospect get OSD for "slow requests are blocked" ? / get detailed health status via librados?
Hi Mohamad, >> I'm currently chewing on an issue regarding "slow requests are blocked". I'd >> like to identify the OSD that is causing those events >> once the cluster is back to HEALTH_OK (as I have no monitoring yet that >> would get this info in realtime). >> >> Collecting this information could help identify aging disks if you were able >> to accumulate and analyze which OSD had blocking >> requests in the past and how often those events occur. >> >> My research so far let's me think that this information is only available as >> long as the requests are actually blocked. Is this >> correct? > > I think this is what you're looking for: > > $> ceph daemon osd.X dump_historic_slow_ops > > which gives you recent slow operations, as opposed to > > $> ceph daemon osd.X dump_blocked_ops > > which returns current blocked operations. You can also add a filter to > those commands. Thanks for these commands. I'll have a look into those. If I understand these correctly it means that I need to run these at each server for each OSD instead of at a central location, is that correct? Regards, Uwe ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] in retrospect get OSD for "slow requests are blocked" ? / get detailed health status via librados?
Hi, On 05/16/2018 04:16 AM, Uwe Sauter wrote: > Hi folks, > > I'm currently chewing on an issue regarding "slow requests are blocked". I'd > like to identify the OSD that is causing those events > once the cluster is back to HEALTH_OK (as I have no monitoring yet that would > get this info in realtime). > > Collecting this information could help identify aging disks if you were able > to accumulate and analyze which OSD had blocking > requests in the past and how often those events occur. > > My research so far let's me think that this information is only available as > long as the requests are actually blocked. Is this > correct? I think this is what you're looking for: $> ceph daemon osd.X dump_historic_slow_ops which gives you recent slow operations, as opposed to $> ceph daemon osd.X dump_blocked_ops which returns current blocked operations. You can also add a filter to those commands. Mohamad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] in retrospect get OSD for "slow requests are blocked" ? / get detailed health status via librados?
Hi folks, I'm currently chewing on an issue regarding "slow requests are blocked". I'd like to identify the OSD that is causing those events once the cluster is back to HEALTH_OK (as I have no monitoring yet that would get this info in realtime). Collecting this information could help identify aging disks if you were able to accumulate and analyze which OSD had blocking requests in the past and how often those events occur. My research so far let's me think that this information is only available as long as the requests are actually blocked. Is this correct? MON logs only show that those events occure and how many requests are in blocking state but no indication of which OSD is affected. Is there a way to identify blocking requests from the OSD log files? On a side note: I was trying to write a small Python script that would extract this kind of information in realtime but while I was able to register a MonitorLog callback that would receive the same messages as you would get with "ceph -w" I haven's seen in the librados Python bindings documentation the possibility to do the equivalent of "ceph health detail". Any suggestions on how to get the blocking OSDs via librados? Thanks, Uwe ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com