Re: [ceph-users] increase pg_num error

[Kyriazis, George]

So, my pool size has increased to a point where the autoscaler did suggest an 
increase of pg_num (from 100 to 512).  Autoscaler mode is “on”, but no change 
happens..

ceph osd pool ls detail reports:
…
pool 10 'rbd1' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins 
pg_num 100 pgp_num 100 pg_num_target 512 pgp_num_target 512 autoscale_mode on 
last_change 3007 flags hashpspool,selfmanaged_snaps,creating stripe_width 0 
application rbd
…

For the pool in question, but no pg resizing activity.  I also tried manually, 
and nothing happens.  No errors in the logs either.

Anybody has any thoughts?  How can I change pg_num (and also pgp_num)?

Thanks!

George


> On Sep 12, 2019, at 7:49 AM, Kyriazis, George  
> wrote:
> 
> Hi Burkhard,
> 
> I tried using the autoscaler, however it did not give a suggestion to resize 
> pg_num.  Since my pg_num is not a power of 2, I wanted to fix that first, 
> manually, to only realize that it didn’t work.
> 
> Because changing pg_num manually did not work, I am not convinced that the 
> autoscaler will work, either, when the time comes.  The autoscaler pg_num 
> changes would follow the same execution path as manual changes, won’t they?
> 
> Thanks!
> 
> George
> 
>> On Sep 12, 2019, at 4:37 AM, Burkhard Linke 
>>  wrote:
>> 
>> Hi,
>> 
>> On 9/12/19 5:16 AM, Kyriazis, George wrote:
>>> Ok, after all is settled, I tried changing pg_num again on my pool and it 
>>> still didn’t work:
>>> 
>>> # ceph osd pool get rbd1 pg_num
>>> pg_num: 100
>>> # ceph osd pool set rbd1 pg_num 128
>>> # ceph osd pool get rbd1 pg_num
>>> pg_num: 100
>>> # ceph osd require-osd-release nautilus
>>> # ceph osd pool set rbd1 pg_num 128
>>> # ceph osd pool get rbd1 pg_num
>>> pg_num: 100
>>> #
>>> 
>>> Suggestions, anybody?
>> 
>> 
>> If you use the autoscaler, changing pg_num will actually change the 
>> pg_num_target setting. The autoscaler will then take care for gently 
>> increasing pg_num until it reaches the value of pg_num_target.
>> 
>> 
>> Regards,
>> 
>> Burkhard
>> 
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] increase pg_num error

[Kyriazis, George]

Hi Burkhard,

I tried using the autoscaler, however it did not give a suggestion to resize 
pg_num.  Since my pg_num is not a power of 2, I wanted to fix that first, 
manually, to only realize that it didn’t work.

Because changing pg_num manually did not work, I am not convinced that the 
autoscaler will work, either, when the time comes.  The autoscaler pg_num 
changes would follow the same execution path as manual changes, won’t they?

Thanks!

George

> On Sep 12, 2019, at 4:37 AM, Burkhard Linke 
>  wrote:
> 
> Hi,
> 
> On 9/12/19 5:16 AM, Kyriazis, George wrote:
>> Ok, after all is settled, I tried changing pg_num again on my pool and it 
>> still didn’t work:
>> 
>> # ceph osd pool get rbd1 pg_num
>> pg_num: 100
>> # ceph osd pool set rbd1 pg_num 128
>> # ceph osd pool get rbd1 pg_num
>> pg_num: 100
>> # ceph osd require-osd-release nautilus
>> # ceph osd pool set rbd1 pg_num 128
>> # ceph osd pool get rbd1 pg_num
>> pg_num: 100
>> #
>> 
>> Suggestions, anybody?
> 
> 
> If you use the autoscaler, changing pg_num will actually change the 
> pg_num_target setting. The autoscaler will then take care for gently 
> increasing pg_num until it reaches the value of pg_num_target.
> 
> 
> Regards,
> 
> Burkhard
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] increase pg_num error

[Burkhard Linke]

Hi,

On 9/12/19 5:16 AM, Kyriazis, George wrote:
Ok, after all is settled, I tried changing pg_num again on my pool and 
it still didn’t work:


# ceph osd pool get rbd1 pg_num
pg_num: 100
# ceph osd pool set rbd1 pg_num 128
# ceph osd pool get rbd1 pg_num
pg_num: 100
# ceph osd require-osd-release nautilus
# ceph osd pool set rbd1 pg_num 128
# ceph osd pool get rbd1 pg_num
pg_num: 100
#

Suggestions, anybody?



If you use the autoscaler, changing pg_num will actually change the 
pg_num_target setting. The autoscaler will then take care for gently 
increasing pg_num until it reaches the value of pg_num_target.



Regards,

Burkhard


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] increase pg_num error

[Kyriazis, George]

Ok, after all is settled, I tried changing pg_num again on my pool and it still 
didn’t work:

# ceph osd pool get rbd1 pg_num
pg_num: 100
# ceph osd pool set rbd1 pg_num 128
# ceph osd pool get rbd1 pg_num
pg_num: 100
# ceph osd require-osd-release nautilus
# ceph osd pool set rbd1 pg_num 128
# ceph osd pool get rbd1 pg_num
pg_num: 100
#

Suggestions, anybody?

Thanks!

George



On Sep 11, 2019, at 5:29 PM, Kyriazis, George 
mailto:george.kyria...@intel.com>> wrote:

No, it’s pg_num first, then pgp_num.

Found the problem, and still slowly working on fixing it.

I upgraded from mimic to nautilus, but forgot to restart the OSD daemons for 2 
of the OSDs.  “ceph osd tell osd.* version” told me which OSDs had a stale 
version.

Then it was just a matter of restarting the osd daemons to bring the version 
up-to-date.  After I did that “ceph -s” was complaining about legacy statfs 
records on OSDs, which means that I had to run “ceph-bluestore-tool repair” on 
the OSDs in question.  That meant taking OSDs “out”, wait for migration, then 
“down”, then stop the OSD daemons, repair, and then reverse the process to 
bring them up.

Now I am waiting for re-mapping to be done, and then I’ll try changing pg_num 
again to see if it works.

Thanks!

George


On Sep 11, 2019, at 5:00 PM, solarflow99 
mailto:solarflo...@gmail.com>> wrote:

You don't have to increase pgp_num first?


On Wed, Sep 11, 2019 at 6:23 AM Kyriazis, George 
mailto:george.kyria...@intel.com>> wrote:
I have the same problem (nautilus installed), but the proposed command gave me 
an error:

# ceph osd require-osd-release nautilus
Error EPERM: not all up OSDs have CEPH_FEATURE_SERVER_NAUTILUS feature
#

I created my cluster with mimic and then upgraded to nautilus.

What would be my next step?

Thanks!

George


> On Jul 1, 2019, at 9:21 AM, Nathan Fish 
> mailto:lordci...@gmail.com>> wrote:
>
> I ran into this recently. Try running "ceph osd require-osd-release
> nautilus". This drops backwards compat with pre-nautilus and allows
> changing settings.
>
> On Mon, Jul 1, 2019 at 4:24 AM Sylvain PORTIER 
> mailto:cab...@free.fr>> wrote:
>>
>> Hi all,
>>
>> I am using ceph 14.2.1 (Nautilus)
>>
>> I am unable to increase the pg_num of a pool.
>>
>> I have a pool named Backup, the current pg_num is 64 : ceph osd pool get
>> Backup pg_num => result pg_num: 64
>>
>> And when I try to increase it using the command
>>
>> ceph osd pool set Backup pg_num 512 => result "set pool 6 pg_num to 512"
>>
>> And then I check with the command : ceph osd pool get Backup pg_num =>
>> result pg_num: 64
>>
>> I don't how to increase the pg_num of a pool, I also tried the autoscale
>> module, but it doesn't work (unable to activate the autoscale, always
>> warn mode).
>>
>> Thank you for your help,
>>
>>
>> Cabeur.
>>
>>
>> ---
>> L'absence de virus dans ce courrier électronique a été vérifiée par le 
>> logiciel antivirus Avast.
>> https://www.avast.com/antivirus
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] increase pg_num error

[Kyriazis, George]

No, it’s pg_num first, then pgp_num.

Found the problem, and still slowly working on fixing it.

I upgraded from mimic to nautilus, but forgot to restart the OSD daemons for 2 
of the OSDs.  “ceph osd tell osd.* version” told me which OSDs had a stale 
version.

Then it was just a matter of restarting the osd daemons to bring the version 
up-to-date.  After I did that “ceph -s” was complaining about legacy statfs 
records on OSDs, which means that I had to run “ceph-bluestore-tool repair” on 
the OSDs in question.  That meant taking OSDs “out”, wait for migration, then 
“down”, then stop the OSD daemons, repair, and then reverse the process to 
bring them up.

Now I am waiting for re-mapping to be done, and then I’ll try changing pg_num 
again to see if it works.

Thanks!

George


On Sep 11, 2019, at 5:00 PM, solarflow99 
mailto:solarflo...@gmail.com>> wrote:

You don't have to increase pgp_num first?


On Wed, Sep 11, 2019 at 6:23 AM Kyriazis, George 
mailto:george.kyria...@intel.com>> wrote:
I have the same problem (nautilus installed), but the proposed command gave me 
an error:

# ceph osd require-osd-release nautilus
Error EPERM: not all up OSDs have CEPH_FEATURE_SERVER_NAUTILUS feature
#

I created my cluster with mimic and then upgraded to nautilus.

What would be my next step?

Thanks!

George


> On Jul 1, 2019, at 9:21 AM, Nathan Fish 
> mailto:lordci...@gmail.com>> wrote:
>
> I ran into this recently. Try running "ceph osd require-osd-release
> nautilus". This drops backwards compat with pre-nautilus and allows
> changing settings.
>
> On Mon, Jul 1, 2019 at 4:24 AM Sylvain PORTIER 
> mailto:cab...@free.fr>> wrote:
>>
>> Hi all,
>>
>> I am using ceph 14.2.1 (Nautilus)
>>
>> I am unable to increase the pg_num of a pool.
>>
>> I have a pool named Backup, the current pg_num is 64 : ceph osd pool get
>> Backup pg_num => result pg_num: 64
>>
>> And when I try to increase it using the command
>>
>> ceph osd pool set Backup pg_num 512 => result "set pool 6 pg_num to 512"
>>
>> And then I check with the command : ceph osd pool get Backup pg_num =>
>> result pg_num: 64
>>
>> I don't how to increase the pg_num of a pool, I also tried the autoscale
>> module, but it doesn't work (unable to activate the autoscale, always
>> warn mode).
>>
>> Thank you for your help,
>>
>>
>> Cabeur.
>>
>>
>> ---
>> L'absence de virus dans ce courrier électronique a été vérifiée par le 
>> logiciel antivirus Avast.
>> https://www.avast.com/antivirus
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] increase pg_num error

[solarflow99]

You don't have to increase pgp_num first?


On Wed, Sep 11, 2019 at 6:23 AM Kyriazis, George 
wrote:

> I have the same problem (nautilus installed), but the proposed command
> gave me an error:
>
> # ceph osd require-osd-release nautilus
> Error EPERM: not all up OSDs have CEPH_FEATURE_SERVER_NAUTILUS feature
> #
>
> I created my cluster with mimic and then upgraded to nautilus.
>
> What would be my next step?
>
> Thanks!
>
> George
>
>
> > On Jul 1, 2019, at 9:21 AM, Nathan Fish  wrote:
> >
> > I ran into this recently. Try running "ceph osd require-osd-release
> > nautilus". This drops backwards compat with pre-nautilus and allows
> > changing settings.
> >
> > On Mon, Jul 1, 2019 at 4:24 AM Sylvain PORTIER  wrote:
> >>
> >> Hi all,
> >>
> >> I am using ceph 14.2.1 (Nautilus)
> >>
> >> I am unable to increase the pg_num of a pool.
> >>
> >> I have a pool named Backup, the current pg_num is 64 : ceph osd pool get
> >> Backup pg_num => result pg_num: 64
> >>
> >> And when I try to increase it using the command
> >>
> >> ceph osd pool set Backup pg_num 512 => result "set pool 6 pg_num to 512"
> >>
> >> And then I check with the command : ceph osd pool get Backup pg_num =>
> >> result pg_num: 64
> >>
> >> I don't how to increase the pg_num of a pool, I also tried the autoscale
> >> module, but it doesn't work (unable to activate the autoscale, always
> >> warn mode).
> >>
> >> Thank you for your help,
> >>
> >>
> >> Cabeur.
> >>
> >>
> >> ---
> >> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> >> https://www.avast.com/antivirus
> >>
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] increase pg_num error

[Kyriazis, George]

I have the same problem (nautilus installed), but the proposed command gave me 
an error:

# ceph osd require-osd-release nautilus
Error EPERM: not all up OSDs have CEPH_FEATURE_SERVER_NAUTILUS feature
# 

I created my cluster with mimic and then upgraded to nautilus.

What would be my next step?

Thanks!

George


> On Jul 1, 2019, at 9:21 AM, Nathan Fish  wrote:
> 
> I ran into this recently. Try running "ceph osd require-osd-release
> nautilus". This drops backwards compat with pre-nautilus and allows
> changing settings.
> 
> On Mon, Jul 1, 2019 at 4:24 AM Sylvain PORTIER  wrote:
>> 
>> Hi all,
>> 
>> I am using ceph 14.2.1 (Nautilus)
>> 
>> I am unable to increase the pg_num of a pool.
>> 
>> I have a pool named Backup, the current pg_num is 64 : ceph osd pool get
>> Backup pg_num => result pg_num: 64
>> 
>> And when I try to increase it using the command
>> 
>> ceph osd pool set Backup pg_num 512 => result "set pool 6 pg_num to 512"
>> 
>> And then I check with the command : ceph osd pool get Backup pg_num =>
>> result pg_num: 64
>> 
>> I don't how to increase the pg_num of a pool, I also tried the autoscale
>> module, but it doesn't work (unable to activate the autoscale, always
>> warn mode).
>> 
>> Thank you for your help,
>> 
>> 
>> Cabeur.
>> 
>> 
>> ---
>> L'absence de virus dans ce courrier électronique a été vérifiée par le 
>> logiciel antivirus Avast.
>> https://www.avast.com/antivirus
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] increase pg_num error

[Robert LeBlanc]

On Mon, Jul 1, 2019 at 11:57 AM Brett Chancellor 
wrote:

> In Nautilus just pg_num is sufficient for both increases and decreases.
>
>
Good to know, I haven't gotten to Nautilus yet.

Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] increase pg_num error

[Brett Chancellor]

In Nautilus just pg_num is sufficient for both increases and decreases.

On Mon, Jul 1, 2019 at 10:55 AM Robert LeBlanc  wrote:

> I believe he needs to increase the pgp_num first, then pg_num.
> 
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>
>
> On Mon, Jul 1, 2019 at 7:21 AM Nathan Fish  wrote:
>
>> I ran into this recently. Try running "ceph osd require-osd-release
>> nautilus". This drops backwards compat with pre-nautilus and allows
>> changing settings.
>>
>> On Mon, Jul 1, 2019 at 4:24 AM Sylvain PORTIER  wrote:
>> >
>> > Hi all,
>> >
>> > I am using ceph 14.2.1 (Nautilus)
>> >
>> > I am unable to increase the pg_num of a pool.
>> >
>> > I have a pool named Backup, the current pg_num is 64 : ceph osd pool get
>> > Backup pg_num => result pg_num: 64
>> >
>> > And when I try to increase it using the command
>> >
>> > ceph osd pool set Backup pg_num 512 => result "set pool 6 pg_num to 512"
>> >
>> > And then I check with the command : ceph osd pool get Backup pg_num =>
>> > result pg_num: 64
>> >
>> > I don't how to increase the pg_num of a pool, I also tried the autoscale
>> > module, but it doesn't work (unable to activate the autoscale, always
>> > warn mode).
>> >
>> > Thank you for your help,
>> >
>> >
>> > Cabeur.
>> >
>> >
>> > ---
>> > L'absence de virus dans ce courrier électronique a été vérifiée par le
>> logiciel antivirus Avast.
>> > https://www.avast.com/antivirus
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] increase pg_num error

[Robert LeBlanc]

I believe he needs to increase the pgp_num first, then pg_num.

Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Mon, Jul 1, 2019 at 7:21 AM Nathan Fish  wrote:

> I ran into this recently. Try running "ceph osd require-osd-release
> nautilus". This drops backwards compat with pre-nautilus and allows
> changing settings.
>
> On Mon, Jul 1, 2019 at 4:24 AM Sylvain PORTIER  wrote:
> >
> > Hi all,
> >
> > I am using ceph 14.2.1 (Nautilus)
> >
> > I am unable to increase the pg_num of a pool.
> >
> > I have a pool named Backup, the current pg_num is 64 : ceph osd pool get
> > Backup pg_num => result pg_num: 64
> >
> > And when I try to increase it using the command
> >
> > ceph osd pool set Backup pg_num 512 => result "set pool 6 pg_num to 512"
> >
> > And then I check with the command : ceph osd pool get Backup pg_num =>
> > result pg_num: 64
> >
> > I don't how to increase the pg_num of a pool, I also tried the autoscale
> > module, but it doesn't work (unable to activate the autoscale, always
> > warn mode).
> >
> > Thank you for your help,
> >
> >
> > Cabeur.
> >
> >
> > ---
> > L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> > https://www.avast.com/antivirus
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] increase pg_num error

[Nathan Fish]

I ran into this recently. Try running "ceph osd require-osd-release
nautilus". This drops backwards compat with pre-nautilus and allows
changing settings.

On Mon, Jul 1, 2019 at 4:24 AM Sylvain PORTIER  wrote:
>
> Hi all,
>
> I am using ceph 14.2.1 (Nautilus)
>
> I am unable to increase the pg_num of a pool.
>
> I have a pool named Backup, the current pg_num is 64 : ceph osd pool get
> Backup pg_num => result pg_num: 64
>
> And when I try to increase it using the command
>
> ceph osd pool set Backup pg_num 512 => result "set pool 6 pg_num to 512"
>
> And then I check with the command : ceph osd pool get Backup pg_num =>
> result pg_num: 64
>
> I don't how to increase the pg_num of a pool, I also tried the autoscale
> module, but it doesn't work (unable to activate the autoscale, always
> warn mode).
>
> Thank you for your help,
>
>
> Cabeur.
>
>
> ---
> L'absence de virus dans ce courrier électronique a été vérifiée par le 
> logiciel antivirus Avast.
> https://www.avast.com/antivirus
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com