May i know what OSD i have to restart in this case? On Wed, Dec 20, 2017 at 9:14 PM David C <[email protected]> wrote:
> You should just need to restart the relavent OSDs for the new backfill > threshold to kick in. > > On 20 Dec 2017 00:14, "Nghia Than" <[email protected]> wrote: > > I added more OSDs few days ago to reduce usage under 70% (nearfull and > full ratio is higher than this value) and it still stuck at > backfill_toofull while rebalance data. > > I tried to change backfill full ratio and it show error (unchangeable) as > below: > > [root@storcp ~]# ceph tell osd.\* injectargs '--osd_backfill_full_ratio > 0.92' > > osd.0: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.1: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.2: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.3: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.4: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.5: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.6: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.7: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.8: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.9: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.10: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.11: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.12: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.13: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.14: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.15: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.16: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.17: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.18: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.19: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.20: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.21: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.22: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.23: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.24: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.25: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.26: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.27: osd_backfill_full_ratio = '0.92' (unchangeable) > > osd.28: osd_backfill_full_ratio = '0.92' (unchangeable) > > [root@storcp ~]# > > On Wed, Dec 20, 2017 at 1:57 AM, David C <[email protected]> wrote: > >> What's your backfill full ratio? You may be able to get healthy by >> increasing your backfill full ratio (in small increments). But your next >> immediate task should be to add more OSDs or remove data. >> >> >> On 19 Dec 2017 4:26 p.m., "Nghia Than" <[email protected]> wrote: >> >> Hi, >> >> My CEPH is stuck at this for few days, we added new OSD and nothing >> changed: >> >> >> - *17 pgs backfill_toofull* >> - *17 pgs stuck unclean* >> - *recovery 21/5156264 objects degraded (0.000%)* >> - *recovery 52908/5156264 objects misplaced (1.026%)* >> - *8 near full osd(s)* >> >> >> And here is my ceph health detail: >> >> HEALTH_WARN 17 pgs backfill_toofull; 17 pgs stuck unclean; recovery >> 21/5156264 objects degraded (0.000%); recovery 52908/5156264 objects >> misplaced (1.026%); 8 near full osd(s) >> >> pg 1.231 is stuck unclean for 4367.088889, current state >> active+remapped+backfill_toofull, last acting [24,9] >> >> pg 1.1e8 is stuck unclean for 7316.364770, current state >> active+remapped+backfill_toofull, last acting [16,3] >> >> pg 1.188 is stuck unclean for 7315.400227, current state >> active+remapped+backfill_toofull, last acting [11,7] >> >> pg 1.158 is stuck unclean for 7321.511627, current state >> active+remapped+backfill_toofull, last acting [11,17] >> >> pg 1.81 is stuck unclean for 4366.683703, current state >> active+remapped+backfill_toofull, last acting [10,24] >> >> pg 1.332 is stuck unclean for 7315.248115, current state >> active+remapped+backfill_toofull, last acting [23,1] >> >> pg 1.2c2 is stuck unclean for 4365.635413, current state >> active+remapped+backfill_toofull, last acting [24,13] >> >> pg 1.3c6 is stuck unclean for 7320.816089, current state >> active+remapped+backfill_toofull, last acting [11,20] >> >> pg 1.26f is stuck unclean for 7315.882215, current state >> active+remapped+backfill_toofull, last acting [28,8] >> >> pg 1.236 is stuck unclean for 7322.152706, current state >> active+remapped+backfill_toofull, last acting [8,26] >> >> pg 1.249 is stuck unclean for 4366.885751, current state >> active+remapped+backfill_toofull, last acting [9,24] >> >> pg 1.7b is stuck unclean for 7315.353072, current state >> active+remapped+backfill_toofull, last acting [28,3] >> >> pg 1.1ec is stuck unclean for 7315.981062, current state >> active+remapped+backfill_toofull, last acting [16,0] >> >> pg 1.248 is stuck unclean for 7324.062482, current state >> active+remapped+backfill_toofull, last acting [16,3] >> >> pg 1.e4 is stuck unclean for 4370.009328, current state >> active+remapped+backfill_toofull, last acting [21,24] >> >> pg 1.144 is stuck unclean for 7317.998393, current state >> active+remapped+backfill_toofull, last acting [26,3] >> >> pg 0.5f is stuck unclean for 5877.987814, current state >> active+remapped+backfill_toofull, last acting [24,5] >> >> pg 1.3c6 is active+remapped+backfill_toofull, acting [11,20] >> >> pg 1.332 is active+remapped+backfill_toofull, acting [23,1] >> >> pg 1.2c2 is active+remapped+backfill_toofull, acting [24,13] >> >> pg 1.26f is active+remapped+backfill_toofull, acting [28,8] >> >> pg 1.249 is active+remapped+backfill_toofull, acting [9,24] >> >> pg 1.248 is active+remapped+backfill_toofull, acting [16,3] >> >> pg 1.236 is active+remapped+backfill_toofull, acting [8,26] >> >> pg 1.e4 is active+remapped+backfill_toofull, acting [21,24] >> >> pg 0.5f is active+remapped+backfill_toofull, acting [24,5] >> >> pg 1.7b is active+remapped+backfill_toofull, acting [28,3] >> >> pg 1.81 is active+remapped+backfill_toofull, acting [10,24] >> >> pg 1.144 is active+remapped+backfill_toofull, acting [26,3] >> >> pg 1.158 is active+remapped+backfill_toofull, acting [11,17] >> >> pg 1.188 is active+remapped+backfill_toofull, acting [11,7] >> >> pg 1.1e8 is active+remapped+backfill_toofull, acting [16,3] >> >> pg 1.1ec is active+remapped+backfill_toofull, acting [16,0] >> >> pg 1.231 is active+remapped+backfill_toofull, acting [24,9] >> >> recovery 21/5156264 objects degraded (0.000%) >> >> recovery 52908/5156264 objects misplaced (1.026%) >> >> osd.3 is near full at 92% >> >> osd.4 is near full at 91% >> >> osd.12 is near full at 92% >> >> osd.17 is near full at 86% >> >> osd.18 is near full at 87% >> >> osd.23 is near full at 90% >> >> osd.27 is near full at 85% >> osd.28 is near full at 85% >> >> I tried reweight OSD to smaller weight but nothing changed. This is my >> dump full_ratio: >> >> [root@storcp ~]# ceph pg dump |grep full_ratio >> >> dumped all in format plain >> >> full_ratio 0.95 >> >> nearfull_ratio 0.85 >> [root@storcp ~]# >> >> And ceph osd df: >> >> [root@storcp ~]# ceph osd df >> >> ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS >> >> 0 0.86800 1.00000 888G 754G 134G 84.91 1.09 102 >> >> 1 0.86800 1.00000 888G 734G 154G 82.63 1.06 90 >> >> 2 0.86800 1.00000 888G 548G 339G 61.77 0.79 75 >> >> 9 0.86800 1.00000 888G 658G 230G 74.09 0.95 81 >> >> 10 0.86800 1.00000 888G 659G 229G 74.17 0.95 79 >> >> 11 0.86800 1.00000 888G 706G 182G 79.49 1.02 91 >> >> 18 0.86800 1.00000 888G 774G 114G 87.14 1.12 94 >> >> 3 0.86800 1.00000 888G 823G 67037M 92.63 1.19 99 >> >> 4 0.86800 1.00000 888G 816G 73780M 91.89 1.18 102 >> >> 5 0.86800 1.00000 888G 608G 279G 68.51 0.88 76 >> >> 12 0.86800 1.00000 888G 818G 72144M 92.07 1.18 111 >> >> 13 0.86800 1.00000 888G 657G 231G 73.94 0.95 84 >> >> 14 0.86800 1.00000 888G 668G 220G 75.16 0.96 84 >> >> 19 0.86800 1.00000 888G 547G 341G 61.59 0.79 75 >> >> 6 0.86800 1.00000 888G 651G 237G 73.31 0.94 90 >> >> 7 0.86800 1.00000 888G 542G 346G 61.01 0.78 68 >> >> 8 0.86800 1.00000 888G 727G 160G 81.90 1.05 90 >> >> 15 0.86800 1.00000 888G 627G 260G 70.64 0.91 91 >> >> 16 0.86800 1.00000 888G 668G 220G 75.19 0.96 81 >> >> 17 0.86800 1.00000 888G 764G 124G 86.04 1.10 92 >> >> 20 0.86800 1.00000 888G 598G 289G 67.37 0.86 78 >> >> 21 0.86800 1.00000 888G 726G 162G 81.73 1.05 87 >> >> 22 0.86800 1.00000 888G 707G 181G 79.60 1.02 92 >> >> 23 0.86800 1.00000 888G 804G 85861M 90.57 1.16 104 >> >> 24 0.86800 0.79999 888G 726G 162G 81.73 1.05 90 >> >> 25 0.86800 1.00000 888G 579G 308G 65.24 0.84 80 >> >> 26 0.86800 1.00000 888G 696G 192G 78.36 1.00 95 >> >> 27 0.86800 1.00000 888G 757G 131G 85.20 1.09 98 >> >> 28 0.86800 1.00000 888G 758G 130G 85.29 1.09 104 >> >> TOTAL 25775G 20115G 5660G 78.04 >> >> MIN/MAX VAR: 0.78/1.19 STDDEV: 9.24 >> [root@storcp ~]# >> >> May i know how to get over this? >> >> -- >> ============== >> Nghia Than >> >> _______________________________________________ >> ceph-users mailing list >> [email protected] >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> > > > -- > ============== > Nghia Than > > > -- ============== Nghia Than
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
