Restarted mgr and mon services, nothing helped :/
-----Original Message----- From: Eugen Block <[email protected]> Sent: Wednesday, May 20, 2020 3:05 PM To: Szabo, Istvan (Agoda) <[email protected]> Cc: [email protected] Subject: [ceph-users] Re: Pool full but the user cleaned it up already Email received from outside the company. If in doubt don't click links nor open attachments! ________________________________ Okay, so the OSDs are in fact not full, it's strange that the pool still is reported as full. Maybe restart the mgr services? Zitat von "Szabo, Istvan (Agoda)" <[email protected]>: > Yeah, sorry: > > ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > 12 ssd 2.29799 1.00000 2.30TiB 2.67GiB 2.30TiB 0.11 0.16 24 > 13 ssd 2.29799 1.00000 2.30TiB 2.33GiB 2.30TiB 0.10 0.14 21 > 14 ssd 3.49300 1.00000 3.49TiB 2.71GiB 3.49TiB 0.08 0.11 27 > 27 ssd 2.29799 1.00000 2.30TiB 2.52GiB 2.30TiB 0.11 0.15 19 > 28 ssd 2.29799 1.00000 2.30TiB 1.79GiB 2.30TiB 0.08 0.11 18 > 29 ssd 3.49300 1.00000 3.49TiB 3.40GiB 3.49TiB 0.10 0.13 35 > 38 ssd 2.29990 1.00000 2.30TiB 3.16GiB 2.30TiB 0.13 0.19 27 > 39 ssd 2.29549 1.00000 2.30TiB 2.05GiB 2.29TiB 0.09 0.12 16 > 40 ssd 3.49300 1.00000 3.49TiB 2.51GiB 3.49TiB 0.07 0.10 29 > 0 hdd 9.09599 1.00000 9.10TiB 15.7GiB 9.08TiB 0.17 0.23 12 > 1 hdd 9.09599 1.00000 9.10TiB 92.5GiB 9.01TiB 0.99 1.38 18 > 2 hdd 9.09599 1.00000 9.10TiB 7.24GiB 9.09TiB 0.08 0.11 15 > 3 hdd 9.09599 1.00000 9.10TiB 17.5GiB 9.08TiB 0.19 0.26 18 > 4 hdd 9.09599 1.00000 9.10TiB 94.4GiB 9.00TiB 1.01 1.41 19 > 5 hdd 9.09599 1.00000 9.10TiB 132GiB 8.97TiB 1.41 1.96 17 > 6 hdd 9.09599 1.00000 9.10TiB 140GiB 8.96TiB 1.51 2.09 22 > 7 hdd 9.09599 1.00000 9.10TiB 15.4GiB 9.08TiB 0.17 0.23 11 > 8 hdd 9.09599 1.00000 9.10TiB 44.7GiB 9.05TiB 0.48 0.67 21 > 9 hdd 9.09599 1.00000 9.10TiB 149GiB 8.95TiB 1.59 2.21 13 > 10 hdd 9.09599 1.00000 9.10TiB 22.5GiB 9.07TiB 0.24 0.34 12 > 11 hdd 9.09599 1.00000 9.10TiB 38.4GiB 9.06TiB 0.41 0.57 22 > 15 hdd 9.09599 1.00000 9.10TiB 99.7GiB 9.00TiB 1.07 1.49 17 > 16 hdd 9.09599 1.00000 9.10TiB 142GiB 8.96TiB 1.52 2.11 23 > 17 hdd 9.09599 1.00000 9.10TiB 23.2GiB 9.07TiB 0.25 0.35 12 > 18 hdd 9.09599 1.00000 9.10TiB 9.44GiB 9.09TiB 0.10 0.14 14 > 19 hdd 9.09599 1.00000 9.10TiB 146GiB 8.95TiB 1.56 2.17 17 > 20 hdd 9.09599 1.00000 9.10TiB 19.8GiB 9.08TiB 0.21 0.30 13 > 21 hdd 9.09599 1.00000 9.10TiB 72.9GiB 9.02TiB 0.78 1.09 12 > 22 hdd 9.09599 1.00000 9.10TiB 80.8GiB 9.02TiB 0.87 1.21 12 > 23 hdd 9.09599 1.00000 9.10TiB 15.8GiB 9.08TiB 0.17 0.24 15 > 24 hdd 9.09599 1.00000 9.10TiB 26.4GiB 9.07TiB 0.28 0.39 20 > 25 hdd 9.09599 1.00000 9.10TiB 92.3GiB 9.01TiB 0.99 1.38 22 > 26 hdd 9.09599 1.00000 9.10TiB 41.2GiB 9.06TiB 0.44 0.61 23 > 30 hdd 9.09560 1.00000 9.10TiB 82.5GiB 9.02TiB 0.89 1.23 19 > 31 hdd 9.09560 1.00000 9.10TiB 162GiB 8.94TiB 1.74 2.42 32 > 32 hdd 9.09560 1.00000 9.10TiB 96.2GiB 9.00TiB 1.03 1.43 19 > 33 hdd 9.09560 1.00000 9.10TiB 102GiB 9.00TiB 1.09 1.52 28 > 34 hdd 9.09560 1.00000 9.10TiB 89.4GiB 9.01TiB 0.96 1.33 23 > 35 hdd 9.09560 1.00000 9.10TiB 36.3GiB 9.06TiB 0.39 0.54 29 > 36 hdd 9.09560 1.00000 9.10TiB 98.5GiB 9.00TiB 1.06 1.47 25 > 37 hdd 9.09560 1.00000 9.10TiB 97.5GiB 9.00TiB 1.05 1.45 25 > TOTAL 315TiB 2.27TiB 313TiB 0.72 MIN/MAX VAR: > 0.10/2.42 STDDEV: 0.54 > > > -----Original Message----- > From: Eugen Block <[email protected]> > Sent: Wednesday, May 20, 2020 1:13 PM > To: Szabo, Istvan (Agoda) <[email protected]> > Cc: [email protected] > Subject: [ceph-users] Re: Pool full but the user cleaned it up already > > Email received from outside the company. If in doubt don't click links > nor open attachments! > ________________________________ > > Please add 'ceph osd df' output, not 'ceph df'. > > > Zitat von "Szabo, Istvan (Agoda)" <[email protected]>: > >> Hello, >> >> No, haven't deleted, this warning is quite long time ago. >> >> ceph health detail >> HEALTH_WARN 1 pool(s) full >> POOL_FULL 1 pool(s) full >> pool 'k8s' is full (no quota) >> ceph df >> GLOBAL: >> SIZE AVAIL RAW USED %RAW USED >> 315TiB 313TiB 2.27TiB 0.72 >> POOLS: >> NAME ID USED %USED MAX >> AVAIL OBJECTS >> k8s 8 200GiB 0.22 90.5TiB 51860 >> >> -----Original Message----- >> From: Eugen Block <[email protected]> >> Sent: Wednesday, May 20, 2020 12:50 PM >> To: [email protected] >> Subject: [ceph-users] Re: Pool full but the user cleaned it up >> already >> >> Email received from outside the company. If in doubt don't click >> links nor open attachments! >> ________________________________ >> >> Hi, >> >> take a look into 'ceph osd df' (maybe share the output) to see which >> OSD(s) are full, they determine if when pool becomes full. >> Did you delete lots of objects from that pool recently? That can take >> some time until the space is really cleared. >> >> >> Zitat von "Szabo, Istvan (Agoda)" <[email protected]>: >> >>> Hi, >>> >>> I have a health warn regarding pool full: >>> >>> health: HEALTH_WARN >>> 1 pool(s) full >>> >>> This is the pool that is complaining: >>> Ceph df: >>> NAME ID USED %USED MAX >>> AVAIL OBJECTS >>> k8s 8 200GiB 0.22 >>> 90.5TiB 51860 >>> >>> Rados df: >>> POOL_NAME USED OBJECTS CLONES COPIES >>> MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR >>> k8s-infraops 200GiB 51860 0 155580 >>> 0 0 0 1256337 126GiB 19012858 944GiB >>> >>> What needs to be cleaned to avoid the pool full? >>> >>> Ceph version luminous 12.2.8. >>> >>> Thank you >>> >>> ________________________________ >>> This message is confidential and is for the sole use of the intended >>> recipient(s). It may also be privileged or otherwise protected by >>> copyright or other legal rules. If you have received it by mistake >>> please let us know by reply email and delete it from your system. It >>> is prohibited to copy this message or disclose its content to anyone. >>> Any confidentiality or privilege is not waived or lost by any >>> mistaken delivery or unauthorized disclosure of the message. All >>> messages sent to and from Agoda may be monitored to ensure >>> compliance with company policies, to protect the company's interests >>> and to remove potential malware. Electronic messages may be >>> intercepted, amended, lost or deleted, or contain viruses. >>> _______________________________________________ >>> ceph-users mailing list -- [email protected] To unsubscribe send an >>> email to [email protected] >> >> >> _______________________________________________ >> ceph-users mailing list -- [email protected] To unsubscribe send an >> email to [email protected] >> >> ________________________________ >> This message is confidential and is for the sole use of the intended >> recipient(s). It may also be privileged or otherwise protected by >> copyright or other legal rules. If you have received it by mistake >> please let us know by reply email and delete it from your system. It >> is prohibited to copy this message or disclose its content to anyone. >> Any confidentiality or privilege is not waived or lost by any >> mistaken delivery or unauthorized disclosure of the message. All >> messages sent to and from Agoda may be monitored to ensure compliance >> with company policies, to protect the company's interests and to >> remove potential malware. Electronic messages may be intercepted, >> amended, lost or deleted, or contain viruses. > > > _______________________________________________ > ceph-users mailing list -- [email protected] To unsubscribe send an > email to [email protected] > > ________________________________ > This message is confidential and is for the sole use of the intended > recipient(s). It may also be privileged or otherwise protected by > copyright or other legal rules. If you have received it by mistake > please let us know by reply email and delete it from your system. It > is prohibited to copy this message or disclose its content to anyone. > Any confidentiality or privilege is not waived or lost by any mistaken > delivery or unauthorized disclosure of the message. All messages sent > to and from Agoda may be monitored to ensure compliance with company > policies, to protect the company's interests and to remove potential > malware. Electronic messages may be intercepted, amended, lost or > deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected] ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
