Do you have quotas enabled on that pool?

Can you also show

ceph df detail


Zitat von "Szabo, Istvan (Agoda)" <istvan.sz...@agoda.com>:

Restarted mgr and mon services, nothing helped :/


-----Original Message-----
From: Eugen Block <ebl...@nde.ag>
Sent: Wednesday, May 20, 2020 3:05 PM
To: Szabo, Istvan (Agoda) <istvan.sz...@agoda.com>
Cc: ceph-users@ceph.io
Subject: [ceph-users] Re: Pool full but the user cleaned it up already

Email received from outside the company. If in doubt don't click links nor open attachments!
________________________________

Okay, so the OSDs are in fact not full, it's strange that the pool still is reported as full. Maybe restart the mgr services?


Zitat von "Szabo, Istvan (Agoda)" <istvan.sz...@agoda.com>:

Yeah, sorry:

ID CLASS WEIGHT  REWEIGHT SIZE    USE     AVAIL   %USE VAR  PGS
12   ssd 2.29799  1.00000 2.30TiB 2.67GiB 2.30TiB 0.11 0.16  24
13   ssd 2.29799  1.00000 2.30TiB 2.33GiB 2.30TiB 0.10 0.14  21
14   ssd 3.49300  1.00000 3.49TiB 2.71GiB 3.49TiB 0.08 0.11  27
27   ssd 2.29799  1.00000 2.30TiB 2.52GiB 2.30TiB 0.11 0.15  19
28   ssd 2.29799  1.00000 2.30TiB 1.79GiB 2.30TiB 0.08 0.11  18
29   ssd 3.49300  1.00000 3.49TiB 3.40GiB 3.49TiB 0.10 0.13  35
38   ssd 2.29990  1.00000 2.30TiB 3.16GiB 2.30TiB 0.13 0.19  27
39   ssd 2.29549  1.00000 2.30TiB 2.05GiB 2.29TiB 0.09 0.12  16
40   ssd 3.49300  1.00000 3.49TiB 2.51GiB 3.49TiB 0.07 0.10  29
 0   hdd 9.09599  1.00000 9.10TiB 15.7GiB 9.08TiB 0.17 0.23  12
 1   hdd 9.09599  1.00000 9.10TiB 92.5GiB 9.01TiB 0.99 1.38  18
 2   hdd 9.09599  1.00000 9.10TiB 7.24GiB 9.09TiB 0.08 0.11  15
 3   hdd 9.09599  1.00000 9.10TiB 17.5GiB 9.08TiB 0.19 0.26  18
 4   hdd 9.09599  1.00000 9.10TiB 94.4GiB 9.00TiB 1.01 1.41  19
 5   hdd 9.09599  1.00000 9.10TiB  132GiB 8.97TiB 1.41 1.96  17
 6   hdd 9.09599  1.00000 9.10TiB  140GiB 8.96TiB 1.51 2.09  22
 7   hdd 9.09599  1.00000 9.10TiB 15.4GiB 9.08TiB 0.17 0.23  11
 8   hdd 9.09599  1.00000 9.10TiB 44.7GiB 9.05TiB 0.48 0.67  21
 9   hdd 9.09599  1.00000 9.10TiB  149GiB 8.95TiB 1.59 2.21  13
10   hdd 9.09599  1.00000 9.10TiB 22.5GiB 9.07TiB 0.24 0.34  12
11   hdd 9.09599  1.00000 9.10TiB 38.4GiB 9.06TiB 0.41 0.57  22
15   hdd 9.09599  1.00000 9.10TiB 99.7GiB 9.00TiB 1.07 1.49  17
16   hdd 9.09599  1.00000 9.10TiB  142GiB 8.96TiB 1.52 2.11  23
17   hdd 9.09599  1.00000 9.10TiB 23.2GiB 9.07TiB 0.25 0.35  12
18   hdd 9.09599  1.00000 9.10TiB 9.44GiB 9.09TiB 0.10 0.14  14
19   hdd 9.09599  1.00000 9.10TiB  146GiB 8.95TiB 1.56 2.17  17
20   hdd 9.09599  1.00000 9.10TiB 19.8GiB 9.08TiB 0.21 0.30  13
21   hdd 9.09599  1.00000 9.10TiB 72.9GiB 9.02TiB 0.78 1.09  12
22   hdd 9.09599  1.00000 9.10TiB 80.8GiB 9.02TiB 0.87 1.21  12
23   hdd 9.09599  1.00000 9.10TiB 15.8GiB 9.08TiB 0.17 0.24  15
24   hdd 9.09599  1.00000 9.10TiB 26.4GiB 9.07TiB 0.28 0.39  20
25   hdd 9.09599  1.00000 9.10TiB 92.3GiB 9.01TiB 0.99 1.38  22
26   hdd 9.09599  1.00000 9.10TiB 41.2GiB 9.06TiB 0.44 0.61  23
30   hdd 9.09560  1.00000 9.10TiB 82.5GiB 9.02TiB 0.89 1.23  19
31   hdd 9.09560  1.00000 9.10TiB  162GiB 8.94TiB 1.74 2.42  32
32   hdd 9.09560  1.00000 9.10TiB 96.2GiB 9.00TiB 1.03 1.43  19
33   hdd 9.09560  1.00000 9.10TiB  102GiB 9.00TiB 1.09 1.52  28
34   hdd 9.09560  1.00000 9.10TiB 89.4GiB 9.01TiB 0.96 1.33  23
35   hdd 9.09560  1.00000 9.10TiB 36.3GiB 9.06TiB 0.39 0.54  29
36   hdd 9.09560  1.00000 9.10TiB 98.5GiB 9.00TiB 1.06 1.47  25
37   hdd 9.09560  1.00000 9.10TiB 97.5GiB 9.00TiB 1.05 1.45  25
                    TOTAL  315TiB 2.27TiB  313TiB 0.72 MIN/MAX VAR:
0.10/2.42  STDDEV: 0.54


-----Original Message-----
From: Eugen Block <ebl...@nde.ag>
Sent: Wednesday, May 20, 2020 1:13 PM
To: Szabo, Istvan (Agoda) <istvan.sz...@agoda.com>
Cc: ceph-users@ceph.io
Subject: [ceph-users] Re: Pool full but the user cleaned it up already

Email received from outside the company. If in doubt don't click links
nor open attachments!
________________________________

Please add 'ceph osd df' output, not 'ceph df'.


Zitat von "Szabo, Istvan (Agoda)" <istvan.sz...@agoda.com>:

Hello,

No, haven't deleted, this warning is quite long time ago.

ceph health detail
HEALTH_WARN 1 pool(s) full
POOL_FULL 1 pool(s) full
    pool 'k8s' is full (no quota)
ceph df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    315TiB     313TiB      2.27TiB          0.72
POOLS:
    NAME                           ID     USED        %USED     MAX
AVAIL     OBJECTS
k8s 8 200GiB 0.22 90.5TiB 51860

-----Original Message-----
From: Eugen Block <ebl...@nde.ag>
Sent: Wednesday, May 20, 2020 12:50 PM
To: ceph-users@ceph.io
Subject: [ceph-users] Re: Pool full but the user cleaned it up
already

Email received from outside the company. If in doubt don't click
links nor open attachments!
________________________________

Hi,

take a look into 'ceph osd df' (maybe share the output) to see which
OSD(s) are full, they determine if when pool becomes full.
Did you delete lots of objects from that pool recently? That can take
some time until the space is really cleared.


Zitat von "Szabo, Istvan (Agoda)" <istvan.sz...@agoda.com>:

Hi,

I have a health warn regarding pool full:

    health: HEALTH_WARN
            1 pool(s) full

This is the pool that is complaining:
Ceph df:
    NAME                           ID     USED        %USED     MAX
AVAIL     OBJECTS
    k8s                                  8       200GiB      0.22
   90.5TiB       51860

Rados df:
POOL_NAME                  USED    OBJECTS CLONES COPIES
MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS      RD      WR_OPS   WR
k8s-infraops                200GiB   51860      0 155580
     0       0        0     1256337  126GiB 19012858  944GiB

What needs to be cleaned to avoid the pool full?

Ceph version luminous 12.2.8.

Thank you

________________________________
This message is confidential and is for the sole use of the intended
recipient(s). It may also be privileged or otherwise protected by
copyright or other legal rules. If you have received it by mistake
please let us know by reply email and delete it from your system. It
is prohibited to copy this message or disclose its content to anyone.
Any confidentiality or privilege is not waived or lost by any
mistaken delivery or unauthorized disclosure of the message. All
messages sent to and from Agoda may be monitored to ensure
compliance with company policies, to protect the company's interests
and to remove potential malware. Electronic messages may be
intercepted, amended, lost or deleted, or contain viruses.
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
email to ceph-users-le...@ceph.io


_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
email to ceph-users-le...@ceph.io

________________________________
This message is confidential and is for the sole use of the intended
recipient(s). It may also be privileged or otherwise protected by
copyright or other legal rules. If you have received it by mistake
please let us know by reply email and delete it from your system. It
is prohibited to copy this message or disclose its content to anyone.
Any confidentiality or privilege is not waived or lost by any
mistaken delivery or unauthorized disclosure of the message. All
messages sent to and from Agoda may be monitored to ensure compliance
with company policies, to protect the company's interests and to
remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.


_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
email to ceph-users-le...@ceph.io

________________________________
This message is confidential and is for the sole use of the intended
recipient(s). It may also be privileged or otherwise protected by
copyright or other legal rules. If you have received it by mistake
please let us know by reply email and delete it from your system. It
is prohibited to copy this message or disclose its content to anyone.
Any confidentiality or privilege is not waived or lost by any mistaken
delivery or unauthorized disclosure of the message. All messages sent
to and from Agoda may be monitored to ensure compliance with company
policies, to protect the company's interests and to remove potential
malware. Electronic messages may be intercepted, amended, lost or
deleted, or contain viruses.


_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.


_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

Reply via email to