On 10/26/19 8:01 AM, Philippe D'Anjou wrote: > V14.2.4 > So, this is not new, this happens every time there is a rebalance, now > because of raising PGs. PG balancer is disabled because I thought it was > the reason but apparently it's not, but it ain't helping either. > > Ceph is totally borged, it's only moving data on nearfull OSDs causing > issues. See this after PG raise. > > health: HEALTH_WARN > 3 nearfull osd(s) > 2 pool(s) nearfull > > > 08:44 am > > 92 hdd 9.09470 1.00000 9.1 TiB 7.5 TiB 7.5 TiB 48 KiB 19 GiB 1.6 > TiB 82.79 1.25 39 up > 71 hdd 9.09470 1.00000 9.1 TiB 7.6 TiB 7.6 TiB 88 KiB 20 GiB 1.4 > TiB 84.09 1.27 38 up > 21 hdd 9.09470 1.00000 9.1 TiB 7.6 TiB 7.6 TiB 60 KiB 20 GiB 1.5 > TiB 84.05 1.27 36 up > > > 08:54 am > > 92 hdd 9.09470 1.00000 9.1 TiB 7.5 TiB 7.5 TiB 48 KiB 19 GiB 1.6 > TiB 82.81 1.25 39 up > 71 hdd 9.09470 1.00000 9.1 TiB 7.7 TiB 7.6 TiB 88 KiB 20 GiB 1.4 > TiB 84.14 1.27 38 up > 21 hdd 9.09470 1.00000 9.1 TiB 7.6 TiB 7.6 TiB 60 KiB 20 GiB 1.4 > TiB 84.10 1.27 36 up > > > > 14 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 76 KiB 17 GiB > 2.6 TiB 71.33 1.07 32 up > 19 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 52 KiB 17 GiB > 2.8 TiB 68.81 1.04 30 up > 22 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 92 KiB 17 GiB > 2.8 TiB 68.90 1.04 32 up > 25 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 219 KiB 17 GiB > 2.9 TiB 68.11 1.03 31 up > 30 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 20 KiB 17 GiB > 2.6 TiB 71.41 1.08 33 up > 33 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 40 KiB 17 GiB > 2.6 TiB 71.30 1.07 32 up > 34 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 36 KiB 17 GiB > 2.6 TiB 71.33 1.07 30 up > 35 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 124 KiB 17 GiB > 2.5 TiB 72.61 1.09 32 up > 12 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 24 KiB 18 GiB > 2.4 TiB 73.84 1.11 32 up > 16 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.4 TiB 96 KiB 17 GiB > 2.6 TiB 71.08 1.07 29 up > 17 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 60 KiB 17 GiB > 2.6 TiB 71.41 1.08 31 up > 20 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 92 KiB 17 GiB > 2.9 TiB 68.57 1.03 28 up > 23 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 36 KiB 17 GiB > 2.6 TiB 71.37 1.08 29 up > 26 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 84 KiB 17 GiB > 2.7 TiB 70.02 1.06 30 up > 28 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 28 KiB 17 GiB > 2.7 TiB 70.11 1.06 30 up > 31 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 56 KiB 17 GiB > 2.6 TiB 71.26 1.07 32 up > 13 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 24 KiB 18 GiB > 2.4 TiB 73.84 1.11 31 up > 15 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 44 KiB 17 GiB > 2.6 TiB 71.35 1.08 29 up > 18 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 76 KiB 16 GiB > 3.3 TiB 63.70 0.96 26 up > 21 hdd 9.09470 1.00000 9.1 TiB 7.6 TiB 7.6 TiB 60 KiB 20 GiB > 1.4 TiB 84.10 1.27 36 up > 24 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 64 KiB 15 GiB > 3.3 TiB 63.67 0.96 28 up > 27 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 48 KiB 17 GiB > 3.1 TiB 66.03 1.00 28 up > 29 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.3 TiB 28 KiB 18 GiB > 2.7 TiB 69.93 1.05 34 up > 32 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 20 KiB 17 GiB > 3.1 TiB 66.20 1.00 28 up > 37 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 32 KiB 18 GiB > 2.7 TiB 70.59 1.06 31 up > 39 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 32 KiB 19 GiB > 2.7 TiB 70.50 1.06 29 up > 41 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 52 KiB 17 GiB > 2.8 TiB 68.79 1.04 30 up > 43 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 48 KiB 17 GiB > 2.8 TiB 68.84 1.04 28 up > 45 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 80 KiB 18 GiB > 2.4 TiB 74.02 1.12 33 up > 46 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 36 KiB 18 GiB > 2.4 TiB 73.88 1.11 30 up > 48 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 101 KiB 17 GiB > 2.5 TiB 72.57 1.09 31 up > 50 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 96 KiB 17 GiB > 2.8 TiB 68.86 1.04 31 up > 36 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 60 KiB 18 GiB > 2.4 TiB 73.82 1.11 32 up > 38 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 76 KiB 17 GiB > 2.7 TiB 70.62 1.06 33 up > 40 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 8 KiB 18 GiB > 2.4 TiB 73.89 1.11 33 up > 42 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 120 KiB 17 GiB > 2.8 TiB 68.82 1.04 28 up > 44 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 68 KiB 17 GiB > 2.6 TiB 71.39 1.08 29 up > 47 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 68 KiB 17 GiB > 2.7 TiB 70.06 1.06 29 up > 49 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 40 KiB 16 GiB > 3.1 TiB 66.25 1.00 28 up > 51 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 60 KiB 15 GiB > 3.3 TiB 63.73 0.96 27 up > 52 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.4 TiB 200 KiB 17 GiB > 2.6 TiB 71.09 1.07 33 up > 53 hdd 9.09470 0.79999 9.1 TiB 7.2 TiB 7.2 TiB 48 KiB 19 GiB > 1.9 TiB 79.03 1.19 33 up > 54 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 44 KiB 15 GiB > 3.5 TiB 61.14 0.92 25 up > 55 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 36 KiB 15 GiB > 3.3 TiB 63.72 0.96 26 up > 56 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 49 KiB 18 GiB > 2.4 TiB 73.91 1.11 31 up > 57 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 72 KiB 17 GiB > 2.8 TiB 68.88 1.04 29 up > 58 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 20 KiB 18 GiB > 2.6 TiB 71.40 1.08 30 up > 59 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 60 KiB 16 GiB > 3.0 TiB 67.51 1.02 29 up > 60 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 32 KiB 16 GiB > 3.1 TiB 66.14 1.00 26 up > 61 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 32 KiB 17 GiB > 2.9 TiB 68.59 1.03 32 up > 62 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 80 KiB 17 GiB > 2.7 TiB 70.06 1.06 29 up > 63 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 76 KiB 18 GiB > 2.4 TiB 73.85 1.11 33 up > 64 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 44 KiB 17 GiB > 2.8 TiB 68.80 1.04 31 up > 65 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 68 KiB 17 GiB > 2.7 TiB 70.10 1.06 32 up > 66 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 52 KiB 17 GiB > 2.8 TiB 68.77 1.04 30 up > 67 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 112 KiB 17 GiB > 2.6 TiB 71.36 1.08 31 up > 84 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 60 KiB 17 GiB > 2.8 TiB 68.84 1.04 30 up > 85 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 6.9 TiB 48 KiB 19 GiB > 2.1 TiB 76.46 1.15 32 up > 86 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 76 KiB 16 GiB > 2.8 TiB 68.80 1.04 29 up > 87 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 60 KiB 16 GiB > 3.1 TiB 66.28 1.00 27 up > 88 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 5.9 TiB 28 KiB 17 GiB > 3.1 TiB 65.51 0.99 27 up > 89 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 116 KiB 17 GiB > 2.8 TiB 68.90 1.04 28 up > 90 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 56 KiB 16 GiB > 3.1 TiB 66.29 1.00 28 up > 91 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 16 KiB 16 GiB > 3.1 TiB 66.24 1.00 30 up > 76 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 76 KiB 17 GiB > 2.8 TiB 68.78 1.04 27 up > 77 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 44 KiB 17 GiB > 2.6 TiB 71.39 1.08 29 up > 78 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 40 KiB 17 GiB > 2.6 TiB 71.38 1.08 32 up > 79 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 56 KiB 17 GiB > 2.6 TiB 71.29 1.07 33 up > 80 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 32 KiB 18 GiB > 2.6 TiB 71.36 1.08 30 up > 81 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 44 KiB 17 GiB > 2.6 TiB 71.36 1.08 32 up > 82 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 36 KiB 17 GiB > 2.8 TiB 68.82 1.04 30 up > 83 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 36 KiB 17 GiB > 2.7 TiB 70.07 1.06 29 up > 68 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 20 KiB 17 GiB > 2.7 TiB 70.06 1.06 31 up > 69 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 16 KiB 18 GiB > 2.4 TiB 73.96 1.11 32 up > 70 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 64 KiB 17 GiB > 2.8 TiB 68.84 1.04 29 up > 71 hdd 9.09470 1.00000 9.1 TiB 7.7 TiB 7.6 TiB 88 KiB 20 GiB > 1.4 TiB 84.14 1.27 38 up > 72 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 48 KiB 16 GiB > 3.0 TiB 67.55 1.02 28 up > 73 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 80 KiB 16 GiB > 3.1 TiB 66.20 1.00 30 up > 74 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 32 KiB 17 GiB > 2.6 TiB 71.25 1.07 31 up > 75 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 44 KiB 16 GiB > 3.1 TiB 66.17 1.00 26 up > 121 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 76 KiB 15 GiB > 3.5 TiB 61.18 0.92 28 up > 123 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 72 KiB 17 GiB > 2.6 TiB 71.47 1.08 29 up > 124 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 76 KiB 17 GiB > 2.6 TiB 71.29 1.07 31 up > 126 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 64 KiB 16 GiB > 2.9 TiB 68.65 1.03 29 up > 128 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 104 KiB 17 GiB > 2.5 TiB 72.65 1.09 32 up > 130 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 96 KiB 16 GiB > 3.1 TiB 66.33 1.00 28 up > 132 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 84 KiB 17 GiB > 2.4 TiB 73.99 1.11 31 up > 134 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.6 TiB 48 KiB 18 GiB > 2.4 TiB 73.22 1.10 30 up > 112 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 64 KiB 18 GiB > 2.4 TiB 73.86 1.11 31 up > 113 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 104 KiB 15 GiB > 3.3 TiB 63.77 0.96 28 up > 114 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 48 KiB 16 GiB > 3.1 TiB 66.29 1.00 28 up > 115 hdd 9.09470 0.89999 9.1 TiB 6.7 TiB 6.7 TiB 40 KiB 18 GiB > 2.4 TiB 73.70 1.11 29 up > 116 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 4 KiB 17 GiB > 2.6 TiB 71.44 1.08 29 up > 117 hdd 9.09470 0.89999 9.1 TiB 6.7 TiB 6.7 TiB 120 KiB 18 GiB > 2.4 TiB 73.90 1.11 29 up > 118 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 16 KiB 15 GiB > 3.4 TiB 62.88 0.95 25 up > 119 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 108 KiB 16 GiB > 2.8 TiB 68.83 1.04 30 up > 120 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 5.9 TiB 68 KiB 16 GiB > 3.1 TiB 65.48 0.99 29 up > 122 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 100 KiB 17 GiB > 2.5 TiB 72.63 1.09 31 up > 125 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 64 KiB 17 GiB > 2.4 TiB 73.85 1.11 32 up > 127 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 171 KiB 17 GiB > 2.5 TiB 72.58 1.09 34 up > 129 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 48 KiB 17 GiB > 2.6 TiB 71.39 1.08 30 up > 131 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 68 KiB 17 GiB > 2.6 TiB 71.38 1.08 29 up > 133 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 28 KiB 17 GiB > 2.4 TiB 73.97 1.11 31 up > 135 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 104 KiB 17 GiB > 2.5 TiB 72.58 1.09 30 up > 92 hdd 9.09470 1.00000 9.1 TiB 7.5 TiB 7.5 TiB 48 KiB 19 GiB > 1.6 TiB 82.81 1.25 39 up > 93 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 88 KiB 16 GiB > 2.8 TiB 68.83 1.04 30 up > 94 hdd 9.09470 1.00000 9.1 TiB 4.9 TiB 4.9 TiB 32 KiB 13 GiB > 4.2 TiB 53.51 0.81 25 up > 95 hdd 8.00000 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 60 KiB 16 GiB > 3.1 TiB 66.28 1.00 27 up > 96 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 76 KiB 15 GiB > 3.3 TiB 63.72 0.96 27 up > 97 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 68 KiB 15 GiB > 3.3 TiB 63.61 0.96 27 up > 98 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 24 KiB 17 GiB > 2.4 TiB 73.82 1.11 29 up > 99 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.4 TiB 76 KiB 17 GiB > 2.6 TiB 71.09 1.07 33 up > > > I really need some help with this, it's becoming unbearable to work > with, I can't attend 24/7 to manually rebalance, this is not how its > supposed to be. This is a Nautilus only issue
This is not an issue, but you seem to be very low on PGs per OSD. This causes CRUSH not to be able to balance in a very optimal way. You have ~30 PGs per OSD on 10TB large OSDs. Try increasing this to 100 ~ 150 PGs or even ~200 per OSD. This will allow for a better data distribution. Wido > > Thanks > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
