Yes I mean the existing Cluster. SSDs are on a fully separate pool. Cluster is not busy during recovery and deep scrubs but I think it’s better to limit replication in some way when switching to replica 3.
My question is to understand if I need to set some options parameters to limit the impact of the creation of new objects.I’m also concerned about disk filling up during recovery because of inefficient data balancing. Here osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -10 19.69994 root ssd -11 5.06998 host ceph101 166 0.98999 osd.166 up 1.00000 1.00000 167 1.00000 osd.167 up 1.00000 1.00000 168 1.00000 osd.168 up 1.00000 1.00000 169 1.07999 osd.169 up 1.00000 1.00000 170 1.00000 osd.170 up 1.00000 1.00000 -12 4.92998 host ceph102 171 0.98000 osd.171 up 1.00000 1.00000 172 0.92999 osd.172 up 1.00000 1.00000 173 0.98000 osd.173 up 1.00000 1.00000 174 1.00000 osd.174 up 1.00000 1.00000 175 1.03999 osd.175 up 1.00000 1.00000 -13 4.69998 host ceph103 176 0.84999 osd.176 up 1.00000 1.00000 177 0.84999 osd.177 up 1.00000 1.00000 178 1.00000 osd.178 up 1.00000 1.00000 179 1.00000 osd.179 up 1.00000 1.00000 180 1.00000 osd.180 up 1.00000 1.00000 -14 5.00000 host ceph104 181 1.00000 osd.181 up 1.00000 1.00000 182 1.00000 osd.182 up 1.00000 1.00000 183 1.00000 osd.183 up 1.00000 1.00000 184 1.00000 osd.184 up 1.00000 1.00000 185 1.00000 osd.185 up 1.00000 1.00000 -1 185.19835 root default -2 18.39980 host ceph001 63 0.79999 osd.63 up 1.00000 1.00000 64 0.79999 osd.64 up 1.00000 1.00000 65 0.79999 osd.65 up 1.00000 1.00000 146 0.70000 osd.146 up 1.00000 1.00000 147 0.70000 osd.147 up 1.00000 1.00000 148 0.90999 osd.148 up 1.00000 1.00000 149 0.79999 osd.149 up 1.00000 1.00000 150 0.79999 osd.150 up 1.00000 1.00000 151 0.79999 osd.151 up 1.00000 1.00000 152 0.79999 osd.152 up 1.00000 1.00000 153 0.70000 osd.153 up 1.00000 1.00000 154 0.70000 osd.154 up 1.00000 1.00000 155 0.89999 osd.155 up 1.00000 1.00000 156 0.84999 osd.156 up 1.00000 1.00000 157 0.70000 osd.157 up 1.00000 1.00000 158 0.70000 osd.158 up 1.00000 1.00000 159 0.84999 osd.159 up 1.00000 1.00000 160 0.90999 osd.160 up 1.00000 1.00000 161 0.90999 osd.161 up 1.00000 1.00000 162 0.90999 osd.162 up 1.00000 1.00000 163 0.79999 osd.163 up 1.00000 1.00000 164 0.90999 osd.164 up 1.00000 1.00000 165 0.64999 osd.165 up 1.00000 1.00000 -3 19.41982 host ceph002 23 0.79999 osd.23 up 1.00000 1.00000 24 0.70000 osd.24 up 1.00000 1.00000 25 0.90999 osd.25 up 1.00000 1.00000 26 0.59999 osd.26 up 1.00000 1.00000 27 0.95000 osd.27 up 1.00000 1.00000 28 0.64999 osd.28 up 1.00000 1.00000 29 0.75000 osd.29 up 1.00000 1.00000 30 0.89999 osd.30 up 1.00000 1.00000 31 0.90999 osd.31 up 1.00000 1.00000 32 0.90999 osd.32 up 1.00000 1.00000 33 0.89999 osd.33 up 1.00000 1.00000 34 0.90999 osd.34 up 1.00000 1.00000 35 0.90999 osd.35 up 1.00000 1.00000 36 0.84999 osd.36 up 1.00000 1.00000 37 0.89999 osd.37 up 1.00000 1.00000 38 1.00000 osd.38 up 1.00000 1.00000 39 0.79999 osd.39 up 1.00000 1.00000 40 0.90999 osd.40 up 1.00000 1.00000 41 0.84999 osd.41 up 1.00000 1.00000 42 0.84999 osd.42 up 1.00000 1.00000 43 0.90999 osd.43 up 1.00000 1.00000 44 0.75000 osd.44 up 1.00000 1.00000 45 0.79999 osd.45 up 1.00000 1.00000 -4 16.78984 host ceph003 20 1.00000 osd.20 up 1.00000 1.00000 21 0.90999 osd.21 up 1.00000 1.00000 22 0.90999 osd.22 up 1.00000 1.00000 46 0.79999 osd.46 up 1.00000 1.00000 47 0.84999 osd.47 up 1.00000 1.00000 48 0.75000 osd.48 up 1.00000 1.00000 49 0.90999 osd.49 up 1.00000 1.00000 50 0.90999 osd.50 up 1.00000 1.00000 51 0.90999 osd.51 up 1.00000 1.00000 52 0.90999 osd.52 up 1.00000 1.00000 53 0.70000 osd.53 up 1.00000 1.00000 54 0.70000 osd.54 up 1.00000 1.00000 55 0.90999 osd.55 up 1.00000 1.00000 56 0.90999 osd.56 up 1.00000 1.00000 57 0.75000 osd.57 up 1.00000 1.00000 58 0.79999 osd.58 up 1.00000 1.00000 59 0.70000 osd.59 up 1.00000 1.00000 60 0.90999 osd.60 up 1.00000 1.00000 61 0.75000 osd.61 up 1.00000 1.00000 62 0.79999 osd.62 up 1.00000 1.00000 -5 16.30986 host ceph004 66 0.95000 osd.66 up 1.00000 1.00000 67 0.79999 osd.67 up 1.00000 1.00000 68 0.84999 osd.68 up 1.00000 1.00000 69 0.84999 osd.69 up 1.00000 1.00000 70 0.70000 osd.70 up 1.00000 1.00000 71 0.75000 osd.71 up 1.00000 1.00000 72 0.64999 osd.72 up 1.00000 1.00000 73 0.90999 osd.73 up 1.00000 1.00000 74 0.75000 osd.74 up 1.00000 1.00000 75 0.81999 osd.75 up 1.00000 1.00000 76 0.90999 osd.76 up 1.00000 1.00000 77 0.79999 osd.77 up 1.00000 1.00000 78 0.59999 osd.78 up 1.00000 1.00000 79 1.00000 osd.79 up 1.00000 1.00000 80 0.90999 osd.80 up 1.00000 1.00000 81 0.79999 osd.81 up 1.00000 1.00000 82 0.89999 osd.82 up 1.00000 1.00000 83 0.70000 osd.83 up 1.00000 1.00000 84 0.75000 osd.84 up 1.00000 1.00000 85 0.90999 osd.85 up 1.00000 1.00000 -6 16.71983 host ceph005 86 0.75000 osd.86 up 1.00000 1.00000 87 0.90999 osd.87 up 1.00000 1.00000 88 0.89999 osd.88 up 1.00000 1.00000 89 0.90999 osd.89 up 1.00000 1.00000 90 0.90999 osd.90 up 1.00000 1.00000 91 0.89999 osd.91 up 1.00000 1.00000 92 0.89999 osd.92 up 1.00000 1.00000 93 0.90999 osd.93 up 1.00000 1.00000 94 0.84999 osd.94 up 1.00000 1.00000 95 0.90999 osd.95 up 1.00000 1.00000 96 0.50000 osd.96 up 1.00000 1.00000 97 0.79999 osd.97 up 1.00000 1.00000 98 0.59999 osd.98 up 1.00000 1.00000 99 0.75000 osd.99 up 1.00000 1.00000 100 1.00000 osd.100 up 1.00000 1.00000 101 0.79999 osd.101 up 1.00000 1.00000 102 0.90999 osd.102 up 1.00000 1.00000 103 0.79999 osd.103 up 1.00000 1.00000 104 0.90999 osd.104 up 1.00000 1.00000 105 0.79999 osd.105 up 1.00000 1.00000 -7 17.19980 host ceph006 106 0.90999 osd.106 up 1.00000 1.00000 107 0.89999 osd.107 up 1.00000 1.00000 108 0.70000 osd.108 up 1.00000 1.00000 109 0.84999 osd.109 up 1.00000 1.00000 110 0.79999 osd.110 up 1.00000 1.00000 111 0.79999 osd.111 up 1.00000 1.00000 112 0.79999 osd.112 up 1.00000 1.00000 113 0.90999 osd.113 up 1.00000 1.00000 115 0.90999 osd.115 up 1.00000 1.00000 116 0.90999 osd.116 up 1.00000 1.00000 117 0.90999 osd.117 up 1.00000 1.00000 118 0.90999 osd.118 up 1.00000 1.00000 119 0.59999 osd.119 up 1.00000 1.00000 120 0.90999 osd.120 up 1.00000 1.00000 121 0.90999 osd.121 up 1.00000 1.00000 122 0.90999 osd.122 up 1.00000 1.00000 123 0.79999 osd.123 up 1.00000 1.00000 124 0.75000 osd.124 up 1.00000 1.00000 125 1.09999 osd.125 up 1.00000 1.00000 114 0.90999 osd.114 up 1.00000 1.00000 -8 16.61984 host ceph007 126 0.90999 osd.126 up 1.00000 1.00000 127 0.75000 osd.127 up 1.00000 1.00000 128 0.84999 osd.128 up 1.00000 1.00000 129 0.90999 osd.129 up 1.00000 1.00000 130 0.70000 osd.130 up 1.00000 1.00000 131 0.78000 osd.131 up 1.00000 1.00000 132 0.79999 osd.132 up 1.00000 1.00000 133 0.54999 osd.133 up 1.00000 1.00000 134 0.79999 osd.134 up 1.00000 1.00000 135 0.87000 osd.135 up 1.00000 1.00000 136 0.90999 osd.136 up 1.00000 1.00000 137 0.90999 osd.137 up 1.00000 1.00000 138 0.90999 osd.138 up 1.00000 1.00000 139 0.89999 osd.139 up 1.00000 1.00000 140 0.75000 osd.140 up 1.00000 1.00000 141 0.75000 osd.141 up 1.00000 1.00000 142 0.90999 osd.142 up 1.00000 1.00000 143 0.84999 osd.143 up 1.00000 1.00000 145 0.90999 osd.145 up 1.00000 1.00000 144 0.89999 osd.144 up 1.00000 1.00000 -9 16.93982 host ceph008 0 0.90999 osd.0 up 1.00000 1.00000 1 0.70000 osd.1 up 1.00000 1.00000 2 0.90999 osd.2 up 1.00000 1.00000 3 0.90999 osd.3 up 1.00000 1.00000 4 0.79999 osd.4 up 1.00000 1.00000 5 0.64999 osd.5 up 1.00000 1.00000 6 0.64999 osd.6 up 1.00000 1.00000 7 0.90999 osd.7 up 1.00000 1.00000 8 0.79999 osd.8 up 1.00000 1.00000 9 0.90999 osd.9 up 1.00000 1.00000 10 0.75000 osd.10 up 1.00000 1.00000 11 0.90999 osd.11 up 1.00000 1.00000 12 0.89999 osd.12 up 1.00000 1.00000 13 0.75000 osd.13 up 1.00000 1.00000 14 1.09999 osd.14 up 1.00000 1.00000 15 0.84999 osd.15 up 1.00000 1.00000 16 0.90999 osd.16 up 1.00000 1.00000 17 0.90999 osd.17 up 1.00000 1.00000 18 0.79999 osd.18 up 1.00000 1.00000 19 0.90999 osd.19 up 1.00000 1.00000 -15 28.79987 host ceph009 186 1.00000 osd.186 up 1.00000 1.00000 187 1.00000 osd.187 up 1.00000 1.00000 188 1.00000 osd.188 up 1.00000 1.00000 189 1.00000 osd.189 up 1.00000 1.00000 190 1.00000 osd.190 up 1.00000 1.00000 191 1.00000 osd.191 up 1.00000 1.00000 192 1.00000 osd.192 up 1.00000 1.00000 193 1.00000 osd.193 up 1.00000 1.00000 194 1.00000 osd.194 up 1.00000 1.00000 195 1.79999 osd.195 up 1.00000 1.00000 196 1.79999 osd.196 up 1.00000 1.00000 197 1.79999 osd.197 up 1.00000 1.00000 198 1.79999 osd.198 up 1.00000 1.00000 199 1.79999 osd.199 up 1.00000 1.00000 200 1.79999 osd.200 up 1.00000 1.00000 201 1.79999 osd.201 up 1.00000 1.00000 202 1.79999 osd.202 up 1.00000 1.00000 203 1.79999 osd.203 up 1.00000 1.00000 204 1.79999 osd.204 up 1.00000 1.00000 205 1.79999 osd.205 up 1.00000 1.00000 -16 17.99988 host ceph010 206 1.79999 osd.206 up 1.00000 1.00000 207 1.79999 osd.207 up 1.00000 1.00000 208 1.79999 osd.208 up 1.00000 1.00000 209 1.79999 osd.209 up 1.00000 1.00000 210 1.79999 osd.210 up 1.00000 1.00000 211 1.79999 osd.211 up 1.00000 1.00000 212 1.79999 osd.212 up 1.00000 1.00000 213 1.79999 osd.213 up 1.00000 1.00000 214 1.79999 osd.214 up 1.00000 1.00000 215 1.79999 osd.215 up 1.00000 1.00000 > Il giorno 20 nov 2017, alle ore 12:17, Christian Balzer <ch...@gol.com> ha > scritto: > > > Hello, > > On Mon, 20 Nov 2017 11:56:31 +0100 Matteo Dacrema wrote: > >> Hi, >> >> I need to switch a cluster of over 200 OSDs from replica 2 to replica 3 > I presume this means the existing cluster and not adding 100 OSDs... > >> There are two different crush maps for HDD and SSDs also mapped to two >> different pools. >> >> Is there a best practice to use? Can this provoke troubles? >> > Are your SSDs a cache-tier or are they a fully separate pool? > > As for troubles, how busy is your cluster during the recovery of failed > OSDs or deep scrubs? > > There are 2 things to consider here: > > 1. The re-balancing and additional replication of all the data, which you > can control/ease by the various knobs present. Ceph version matters to > which are relevant/useful. It shouldn't impact things too much, unless > your cluster was at the very edge of it's capacity anyway. > > 2. The little detail that after 1) is done, your cluster will be > noticeably slower than before, especially in the latency department. > In short, you don't just need to have the disk space to go 3x, but also > enough IOPS/bandwidth reserves. > > Christian > >> Thank you >> Matteo >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > -- > Christian Balzer Network/Systems Engineer > ch...@gol.com Rakuten Communications > > -- > Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato non > infetto. > Seguire il link qui sotto per segnalarlo come spam: > http://mx01.enter.it/cgi-bin/learn-msg.cgi?id=524464756E.A33EC > > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
