subject:"\\\\\\\[ceph\\\\\\\-users\\\\\\\] Switch to replica 3"

Re: [ceph-users] Switch to replica 3

2017-11-20 Thread Matteo Dacrema

Ok, thank you guys

The version is 10.2.10

Matteo

> Il giorno 20 nov 2017, alle ore 23:15, Christian Balzer  ha 
> scritto:
> 
> On Mon, 20 Nov 2017 10:35:36 -0800 Chris Taylor wrote:
> 
>> On 2017-11-20 3:39 am, Matteo Dacrema wrote:
>>> Yes I mean the existing Cluster.
>>> SSDs are on a fully separate pool.
>>> Cluster is not busy during recovery and deep scrubs but I think it’s
>>> better to limit replication in some way when switching to replica 3.
>>> 
>>> My question is to understand if I need to set some options parameters
>>> to limit the impact of the creation of new objects.I’m also concerned
>>> about disk filling up during recovery because of inefficient data
>>> balancing.  
>> 
>> You can try using osd_recovery_sleep to slow down the backfilling so it 
>> does not cause the client io to hang.
>> 
>> ceph tell osd.* injectargs "--osd_recovery_sleep 0.1"
>> 
> 
> Which is one of the things that is version specific and we don't know the
> version yet.
> 
> The above will work with Hammer and should again with Luminous, but not so
> much with the unified queue bits inbetween. 
> 
> Christian
> 
>> 
>>> 
>>> Here osd tree
>>> 
>>> ID  WEIGHTTYPE NAMEUP/DOWN REWEIGHT PRIMARY-AFFINITY
>>> -10  19.69994 root ssd
>>> -11   5.06998 host ceph101
>>> 166   0.98999 osd.166   up  1.0  1.0
>>> 167   1.0 osd.167   up  1.0  1.0
>>> 168   1.0 osd.168   up  1.0  1.0
>>> 169   1.07999 osd.169   up  1.0  1.0
>>> 170   1.0 osd.170   up  1.0  1.0
>>> -12   4.92998 host ceph102
>>> 171   0.98000 osd.171   up  1.0  1.0
>>> 172   0.92999 osd.172   up  1.0  1.0
>>> 173   0.98000 osd.173   up  1.0  1.0
>>> 174   1.0 osd.174   up  1.0  1.0
>>> 175   1.03999 osd.175   up  1.0  1.0
>>> -13   4.69998 host ceph103
>>> 176   0.84999 osd.176   up  1.0  1.0
>>> 177   0.84999 osd.177   up  1.0  1.0
>>> 178   1.0 osd.178   up  1.0  1.0
>>> 179   1.0 osd.179   up  1.0  1.0
>>> 180   1.0 osd.180   up  1.0  1.0
>>> -14   5.0 host ceph104
>>> 181   1.0 osd.181   up  1.0  1.0
>>> 182   1.0 osd.182   up  1.0  1.0
>>> 183   1.0 osd.183   up  1.0  1.0
>>> 184   1.0 osd.184   up  1.0  1.0
>>> 185   1.0 osd.185   up  1.0  1.0
>>> -1 185.19835 root default
>>> -2  18.39980 host ceph001
>>> 63   0.7 osd.63up  1.0  1.0
>>> 64   0.7 osd.64up  1.0  1.0
>>> 65   0.7 osd.65up  1.0  1.0
>>> 146   0.7 osd.146   up  1.0  1.0
>>> 147   0.7 osd.147   up  1.0  1.0
>>> 148   0.90999 osd.148   up  1.0  1.0
>>> 149   0.7 osd.149   up  1.0  1.0
>>> 150   0.7 osd.150   up  1.0  1.0
>>> 151   0.7 osd.151   up  1.0  1.0
>>> 152   0.7 osd.152   up  1.0  1.0
>>> 153   0.7 osd.153   up  1.0  1.0
>>> 154   0.7 osd.154   up  1.0  1.0
>>> 155   0.8 osd.155   up  1.0  1.0
>>> 156   0.84999 osd.156   up  1.0  1.0
>>> 157   0.7 osd.157   up  1.0  1.0
>>> 158   0.7 osd.158   up  1.0  1.0
>>> 159   0.84999 osd.159   up  1.0  1.0
>>> 160   0.90999 osd.160   up  1.0  1.0
>>> 161   0.90999 osd.161   up  1.0  1.0
>>> 162   0.90999 osd.162   up  1.0  1.0
>>> 163   0.7 osd.163   up  1.0  1.0
>>> 164   0.90999 osd.164   up  1.0  1.0
>>> 165   0.64999 osd.165   up  1.0  1.0
>>> -3  19.41982 host ceph002
>>> 23   0.7 osd.23up  1.0  1.0
>>> 24   0.7 osd.24up  1.0  1.0
>>> 25   0.90999 osd.25up  1.0  1.0
>>> 26   0.5 osd.26up  1.0  1.0
>>> 27   0.95000 osd.27up  1.0  1.0
>>> 28   0.64999 osd.28up  1.0  1.0
>>> 29   0.75000 osd.29up  1.0  1.0
>>> 30   0.8 osd.30up  1.0

Re: [ceph-users] Switch to replica 3

2017-11-20 Thread Christian Balzer

On Mon, 20 Nov 2017 10:35:36 -0800 Chris Taylor wrote:

> On 2017-11-20 3:39 am, Matteo Dacrema wrote:
> > Yes I mean the existing Cluster.
> > SSDs are on a fully separate pool.
> > Cluster is not busy during recovery and deep scrubs but I think it’s
> > better to limit replication in some way when switching to replica 3.
> > 
> > My question is to understand if I need to set some options parameters
> > to limit the impact of the creation of new objects.I’m also concerned
> > about disk filling up during recovery because of inefficient data
> > balancing.  
> 
> You can try using osd_recovery_sleep to slow down the backfilling so it 
> does not cause the client io to hang.
> 
> ceph tell osd.* injectargs "--osd_recovery_sleep 0.1"
> 

Which is one of the things that is version specific and we don't know the
version yet.

The above will work with Hammer and should again with Luminous, but not so
much with the unified queue bits inbetween. 

Christian

> 
> > 
> > Here osd tree
> > 
> > ID  WEIGHTTYPE NAMEUP/DOWN REWEIGHT PRIMARY-AFFINITY
> > -10  19.69994 root ssd
> > -11   5.06998 host ceph101
> > 166   0.98999 osd.166   up  1.0  1.0
> > 167   1.0 osd.167   up  1.0  1.0
> > 168   1.0 osd.168   up  1.0  1.0
> > 169   1.07999 osd.169   up  1.0  1.0
> > 170   1.0 osd.170   up  1.0  1.0
> > -12   4.92998 host ceph102
> > 171   0.98000 osd.171   up  1.0  1.0
> > 172   0.92999 osd.172   up  1.0  1.0
> > 173   0.98000 osd.173   up  1.0  1.0
> > 174   1.0 osd.174   up  1.0  1.0
> > 175   1.03999 osd.175   up  1.0  1.0
> > -13   4.69998 host ceph103
> > 176   0.84999 osd.176   up  1.0  1.0
> > 177   0.84999 osd.177   up  1.0  1.0
> > 178   1.0 osd.178   up  1.0  1.0
> > 179   1.0 osd.179   up  1.0  1.0
> > 180   1.0 osd.180   up  1.0  1.0
> > -14   5.0 host ceph104
> > 181   1.0 osd.181   up  1.0  1.0
> > 182   1.0 osd.182   up  1.0  1.0
> > 183   1.0 osd.183   up  1.0  1.0
> > 184   1.0 osd.184   up  1.0  1.0
> > 185   1.0 osd.185   up  1.0  1.0
> >  -1 185.19835 root default
> >  -2  18.39980 host ceph001
> >  63   0.7 osd.63up  1.0  1.0
> >  64   0.7 osd.64up  1.0  1.0
> >  65   0.7 osd.65up  1.0  1.0
> > 146   0.7 osd.146   up  1.0  1.0
> > 147   0.7 osd.147   up  1.0  1.0
> > 148   0.90999 osd.148   up  1.0  1.0
> > 149   0.7 osd.149   up  1.0  1.0
> > 150   0.7 osd.150   up  1.0  1.0
> > 151   0.7 osd.151   up  1.0  1.0
> > 152   0.7 osd.152   up  1.0  1.0
> > 153   0.7 osd.153   up  1.0  1.0
> > 154   0.7 osd.154   up  1.0  1.0
> > 155   0.8 osd.155   up  1.0  1.0
> > 156   0.84999 osd.156   up  1.0  1.0
> > 157   0.7 osd.157   up  1.0  1.0
> > 158   0.7 osd.158   up  1.0  1.0
> > 159   0.84999 osd.159   up  1.0  1.0
> > 160   0.90999 osd.160   up  1.0  1.0
> > 161   0.90999 osd.161   up  1.0  1.0
> > 162   0.90999 osd.162   up  1.0  1.0
> > 163   0.7 osd.163   up  1.0  1.0
> > 164   0.90999 osd.164   up  1.0  1.0
> > 165   0.64999 osd.165   up  1.0  1.0
> >  -3  19.41982 host ceph002
> >  23   0.7 osd.23up  1.0  1.0
> >  24   0.7 osd.24up  1.0  1.0
> >  25   0.90999 osd.25up  1.0  1.0
> >  26   0.5 osd.26up  1.0  1.0
> >  27   0.95000 osd.27up  1.0  1.0
> >  28   0.64999 osd.28up  1.0  1.0
> >  29   0.75000 osd.29up  1.0  1.0
> >  30   0.8 osd.30up  1.0  1.0
> >  31   0.90999 osd.31up  1.0  1.0
> >  32   0.90999 osd.32up  1.0  1.0
> >  33

Re: [ceph-users] Switch to replica 3

2017-11-20 Thread Chris Taylor



On 2017-11-20 3:39 am, Matteo Dacrema wrote:

Yes I mean the existing Cluster.
SSDs are on a fully separate pool.
Cluster is not busy during recovery and deep scrubs but I think it’s
better to limit replication in some way when switching to replica 3.

My question is to understand if I need to set some options parameters
to limit the impact of the creation of new objects.I’m also concerned
about disk filling up during recovery because of inefficient data
balancing.


You can try using osd_recovery_sleep to slow down the backfilling so it 
does not cause the client io to hang.


ceph tell osd.* injectargs "--osd_recovery_sleep 0.1"




Here osd tree

ID  WEIGHTTYPE NAMEUP/DOWN REWEIGHT PRIMARY-AFFINITY
-10  19.69994 root ssd
-11   5.06998 host ceph101
166   0.98999 osd.166   up  1.0  1.0
167   1.0 osd.167   up  1.0  1.0
168   1.0 osd.168   up  1.0  1.0
169   1.07999 osd.169   up  1.0  1.0
170   1.0 osd.170   up  1.0  1.0
-12   4.92998 host ceph102
171   0.98000 osd.171   up  1.0  1.0
172   0.92999 osd.172   up  1.0  1.0
173   0.98000 osd.173   up  1.0  1.0
174   1.0 osd.174   up  1.0  1.0
175   1.03999 osd.175   up  1.0  1.0
-13   4.69998 host ceph103
176   0.84999 osd.176   up  1.0  1.0
177   0.84999 osd.177   up  1.0  1.0
178   1.0 osd.178   up  1.0  1.0
179   1.0 osd.179   up  1.0  1.0
180   1.0 osd.180   up  1.0  1.0
-14   5.0 host ceph104
181   1.0 osd.181   up  1.0  1.0
182   1.0 osd.182   up  1.0  1.0
183   1.0 osd.183   up  1.0  1.0
184   1.0 osd.184   up  1.0  1.0
185   1.0 osd.185   up  1.0  1.0
 -1 185.19835 root default
 -2  18.39980 host ceph001
 63   0.7 osd.63up  1.0  1.0
 64   0.7 osd.64up  1.0  1.0
 65   0.7 osd.65up  1.0  1.0
146   0.7 osd.146   up  1.0  1.0
147   0.7 osd.147   up  1.0  1.0
148   0.90999 osd.148   up  1.0  1.0
149   0.7 osd.149   up  1.0  1.0
150   0.7 osd.150   up  1.0  1.0
151   0.7 osd.151   up  1.0  1.0
152   0.7 osd.152   up  1.0  1.0
153   0.7 osd.153   up  1.0  1.0
154   0.7 osd.154   up  1.0  1.0
155   0.8 osd.155   up  1.0  1.0
156   0.84999 osd.156   up  1.0  1.0
157   0.7 osd.157   up  1.0  1.0
158   0.7 osd.158   up  1.0  1.0
159   0.84999 osd.159   up  1.0  1.0
160   0.90999 osd.160   up  1.0  1.0
161   0.90999 osd.161   up  1.0  1.0
162   0.90999 osd.162   up  1.0  1.0
163   0.7 osd.163   up  1.0  1.0
164   0.90999 osd.164   up  1.0  1.0
165   0.64999 osd.165   up  1.0  1.0
 -3  19.41982 host ceph002
 23   0.7 osd.23up  1.0  1.0
 24   0.7 osd.24up  1.0  1.0
 25   0.90999 osd.25up  1.0  1.0
 26   0.5 osd.26up  1.0  1.0
 27   0.95000 osd.27up  1.0  1.0
 28   0.64999 osd.28up  1.0  1.0
 29   0.75000 osd.29up  1.0  1.0
 30   0.8 osd.30up  1.0  1.0
 31   0.90999 osd.31up  1.0  1.0
 32   0.90999 osd.32up  1.0  1.0
 33   0.8 osd.33up  1.0  1.0
 34   0.90999 osd.34up  1.0  1.0
 35   0.90999 osd.35up  1.0  1.0
 36   0.84999 osd.36up  1.0  1.0
 37   0.8 osd.37up  1.0  1.0
 38   1.0 osd.38up  1.0  1.0
 39   0.7 osd.39up  1.0  1.0
 40   0.90999 osd.40up  1.0  1.0
 41   0.84999 osd.41up  1.0  1.0
 42

Re: [ceph-users] Switch to replica 3

2017-11-20 Thread Matteo Dacrema

Yes I mean the existing Cluster.
SSDs are on a fully separate pool.
Cluster is not busy during recovery and deep scrubs but I think it’s better to 
limit replication in some way when switching to replica 3.

My question is to understand if I need to set some options parameters to limit 
the impact of the creation of new objects.I’m also concerned about disk filling 
up during recovery because of inefficient data balancing.

Here osd tree

ID  WEIGHTTYPE NAMEUP/DOWN REWEIGHT PRIMARY-AFFINITY
-10  19.69994 root ssd
-11   5.06998 host ceph101
166   0.98999 osd.166   up  1.0  1.0
167   1.0 osd.167   up  1.0  1.0
168   1.0 osd.168   up  1.0  1.0
169   1.07999 osd.169   up  1.0  1.0
170   1.0 osd.170   up  1.0  1.0
-12   4.92998 host ceph102
171   0.98000 osd.171   up  1.0  1.0
172   0.92999 osd.172   up  1.0  1.0
173   0.98000 osd.173   up  1.0  1.0
174   1.0 osd.174   up  1.0  1.0
175   1.03999 osd.175   up  1.0  1.0
-13   4.69998 host ceph103
176   0.84999 osd.176   up  1.0  1.0
177   0.84999 osd.177   up  1.0  1.0
178   1.0 osd.178   up  1.0  1.0
179   1.0 osd.179   up  1.0  1.0
180   1.0 osd.180   up  1.0  1.0
-14   5.0 host ceph104
181   1.0 osd.181   up  1.0  1.0
182   1.0 osd.182   up  1.0  1.0
183   1.0 osd.183   up  1.0  1.0
184   1.0 osd.184   up  1.0  1.0
185   1.0 osd.185   up  1.0  1.0
 -1 185.19835 root default
 -2  18.39980 host ceph001
 63   0.7 osd.63up  1.0  1.0
 64   0.7 osd.64up  1.0  1.0
 65   0.7 osd.65up  1.0  1.0
146   0.7 osd.146   up  1.0  1.0
147   0.7 osd.147   up  1.0  1.0
148   0.90999 osd.148   up  1.0  1.0
149   0.7 osd.149   up  1.0  1.0
150   0.7 osd.150   up  1.0  1.0
151   0.7 osd.151   up  1.0  1.0
152   0.7 osd.152   up  1.0  1.0
153   0.7 osd.153   up  1.0  1.0
154   0.7 osd.154   up  1.0  1.0
155   0.8 osd.155   up  1.0  1.0
156   0.84999 osd.156   up  1.0  1.0
157   0.7 osd.157   up  1.0  1.0
158   0.7 osd.158   up  1.0  1.0
159   0.84999 osd.159   up  1.0  1.0
160   0.90999 osd.160   up  1.0  1.0
161   0.90999 osd.161   up  1.0  1.0
162   0.90999 osd.162   up  1.0  1.0
163   0.7 osd.163   up  1.0  1.0
164   0.90999 osd.164   up  1.0  1.0
165   0.64999 osd.165   up  1.0  1.0
 -3  19.41982 host ceph002
 23   0.7 osd.23up  1.0  1.0
 24   0.7 osd.24up  1.0  1.0
 25   0.90999 osd.25up  1.0  1.0
 26   0.5 osd.26up  1.0  1.0
 27   0.95000 osd.27up  1.0  1.0
 28   0.64999 osd.28up  1.0  1.0
 29   0.75000 osd.29up  1.0  1.0
 30   0.8 osd.30up  1.0  1.0
 31   0.90999 osd.31up  1.0  1.0
 32   0.90999 osd.32up  1.0  1.0
 33   0.8 osd.33up  1.0  1.0
 34   0.90999 osd.34up  1.0  1.0
 35   0.90999 osd.35up  1.0  1.0
 36   0.84999 osd.36up  1.0  1.0
 37   0.8 osd.37up  1.0  1.0
 38   1.0 osd.38up  1.0  1.0
 39   0.7 osd.39up  1.0  1.0
 40   0.90999 osd.40up  1.0  1.0
 41   0.84999 osd.41up  1.0  1.0
 42   0.84999 osd.42up  1.0  1.0
 43   0.90999 osd.43up  1.0  1.0
 44   0.75000 osd.44up  1.0  1.0
 45   0.7 osd.45

Re: [ceph-users] Switch to replica 3

2017-11-20 Thread Christian Balzer

Hello,

On Mon, 20 Nov 2017 11:56:31 +0100 Matteo Dacrema wrote:

> Hi,
> 
> I need to switch a cluster of over 200 OSDs from replica 2 to replica 3
I presume this means the existing cluster and not adding 100 OSDs...

> There are two different crush maps for HDD and SSDs also mapped to two 
> different pools.
>
> Is there a best practice to use? Can this provoke troubles?
> 
Are your SSDs a cache-tier or are they a fully separate pool?

As for troubles, how busy is your cluster during the recovery of failed
OSDs or deep scrubs?

There are 2 things to consider here:

1. The re-balancing and additional replication of all the data, which you
can control/ease by the various knobs present. Ceph version matters to
which are relevant/useful. It shouldn't impact things too much, unless
your cluster was at the very edge of it's capacity anyway.

2. The little detail that after 1) is done, your cluster will be
noticeably slower than before, especially in the latency department. 
In short, you don't just need to have the disk space to go 3x, but also
enough IOPS/bandwidth reserves.

Christian

> Thank you
> Matteo
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Rakuten Communications
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Switch to replica 3

2017-11-20 Thread Wido den Hollander


> Op 20 november 2017 om 11:56 schreef Matteo Dacrema :
> 
> 
> Hi,
> 
> I need to switch a cluster of over 200 OSDs from replica 2 to replica 3
> There are two different crush maps for HDD and SSDs also mapped to two 
> different pools.
> 
> Is there a best practice to use? Can this provoke troubles?
> 

The command is very simple, but without more information nobody can tell you.

Can you should (attach) the output of 'ceph osd tree'?

Wido

> Thank you
> Matteo
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Switch to replica 3

2017-11-20 Thread Matteo Dacrema

Hi,

I need to switch a cluster of over 200 OSDs from replica 2 to replica 3
There are two different crush maps for HDD and SSDs also mapped to two 
different pools.

Is there a best practice to use? Can this provoke troubles?

Thank you
Matteo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Switch to replica 3

Re: [ceph-users] Switch to replica 3

Re: [ceph-users] Switch to replica 3

Re: [ceph-users] Switch to replica 3

Re: [ceph-users] Switch to replica 3

Re: [ceph-users] Switch to replica 3

[ceph-users] Switch to replica 3

7 matches

Site Navigation

Mail list logo

Footer information