[ceph-users] Re: Experience reducing size 3 to 2 on production cluster?

[Marco Pizzolo]

Thanks Linh Vu, so it sounds like i should be prepared to bounce the OSDs
and/or Hosts, but I haven't heard anyone yet say that it won't work, so I
guess there's that...

On Tue, Dec 14, 2021 at 7:48 PM Linh Vu  wrote:

> I haven't tested this in Nautilus 14.2.22 (or any nautilus) but in
> Luminous or older, if you go from a bigger size to a smaller size, there
> was either a bug or a "feature-not-bug" that didn't allow the OSDs to
> automatically purge the redundant PGs with data copies. I did this on a
> size=5 to size=3 situation in a 1000+ OSD cluster, and also just recently
> in a test Luminous cluster (size=3 to size=2). In order for the purge to
> actually happen, I had to restart every OSD (one at a time for safety, or
> just run ceph-ansible site.yml with the osd handler health check = true).
>
> On Wed, Dec 15, 2021 at 8:58 AM Marco Pizzolo 
> wrote:
>
>> Hi Martin,
>>
>> Agreed on the min_size of 2.  I have no intention of worrying about uptime
>> in event of a host failure.  Once size of 2 is effectuated (and I'm unsure
>> how long it will take), it is our intention to evacuate all OSDs in one of
>> 4 hosts, in order to migrate the host to the new cluster, where its OSDs
>> will then be added in.  Once added and balanced, we will complete the
>> copies (<3 days) and then migrate one more host allowing us to bring size
>> to 3.  Once balanced, we will collapse the last 2 nodes into the new
>> cluster.  I am hoping that inclusive of rebalancing the whole project will
>> only take 3 weeks, but time will tell.
>>
>> Has anyone asked Ceph to reduce hundreds of millions if not billions of
>> files from size 3 to size 2, and if so, were you successful?  I know it
>> *should* be able to do this, but sometimes theory and practice don't
>> perfectly overlap.
>>
>> Thanks,
>> Marco
>>
>> On Sat, Dec 11, 2021 at 4:37 AM Martin Verges 
>> wrote:
>>
>> > Hello,
>> >
>> > avoid size 2 whenever you can. As long as you know that you might lose
>> > data, it can be an acceptable risk while migrating the cluster. We had
>> that
>> > in the past multiple time and it is a valid use case in our opinion.
>> > However make sure to monitor the state and recover as fast as possible.
>> > Leave min_size on 2 as well and accept the potential downtime!
>> >
>> > --
>> > Martin Verges
>> > Managing director
>> >
>> > Mobile: +49 174 9335695  | Chat: https://t.me/MartinVerges
>> >
>> > croit GmbH, Freseniusstr. 31h, 81247 Munich
>> > CEO: Martin Verges - VAT-ID: DE310638492
>> > Com. register: Amtsgericht Munich HRB 231263
>> > Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
>> >
>> >
>> > On Fri, 10 Dec 2021 at 18:05, Marco Pizzolo 
>> > wrote:
>> >
>> >> Hello,
>> >>
>> >> As part of a migration process where we will be swinging Ceph hosts
>> from
>> >> one cluster to another we need to reduce the size from 3 to 2 in order
>> to
>> >> shrink the footprint sufficiently to allow safe removal of an OSD/Mon
>> >> node.
>> >>
>> >> The cluster has about 500M objects as per dashboard, and is about
>> 1.5PB in
>> >> size comprised solely of small files served through CephFS to Samba.
>> >>
>> >> Has anyone encountered a similar situation?  What (if any) problems did
>> >> you
>> >> face?
>> >>
>> >> Ceph 14.2.22 bare metal deployment on Centos.
>> >>
>> >> Thanks in advance.
>> >>
>> >> Marco
>> >> ___
>> >> ceph-users mailing list -- ceph-users@ceph.io
>> >> To unsubscribe send an email to ceph-users-le...@ceph.io
>> >>
>> >
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Experience reducing size 3 to 2 on production cluster?

[Marco Pizzolo]

Hi Joachim,

Understood on the risks.  Aside from the alt. cluster, we have 3 other
copies of the data outside of Ceph, so I feel pretty confident that it's a
question of time to repopulate and not data loss.

That said, I would be interested in your experience on what I'm trying to
do if you've attempted something similar previously.

Thanks,
Marco

On Sat, Dec 11, 2021 at 6:59 AM Joachim Kraftmayer (Clyso GmbH) <
joachim.kraftma...@clyso.com> wrote:

> Hi Marco,
>
> to quote an old colleague, this is one of the ways to break a Ceph
> cluster with its data.
>
> Perhaps the risks are not immediately visible in normal operation, but
> in the event of a failure, the potential loss of data must be accepted.
>
> Regards, Joachim
>
>
> ___
>
> Clyso GmbH - ceph foundation member
>
> Am 10.12.21 um 18:04 schrieb Marco Pizzolo:
> > Hello,
> >
> > As part of a migration process where we will be swinging Ceph hosts from
> > one cluster to another we need to reduce the size from 3 to 2 in order to
> > shrink the footprint sufficiently to allow safe removal of an OSD/Mon
> node.
> >
> > The cluster has about 500M objects as per dashboard, and is about 1.5PB
> in
> > size comprised solely of small files served through CephFS to Samba.
> >
> > Has anyone encountered a similar situation?  What (if any) problems did
> you
> > face?
> >
> > Ceph 14.2.22 bare metal deployment on Centos.
> >
> > Thanks in advance.
> >
> > Marco
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Experience reducing size 3 to 2 on production cluster?

[Marco Pizzolo]

Hi Martin,

Agreed on the min_size of 2.  I have no intention of worrying about uptime
in event of a host failure.  Once size of 2 is effectuated (and I'm unsure
how long it will take), it is our intention to evacuate all OSDs in one of
4 hosts, in order to migrate the host to the new cluster, where its OSDs
will then be added in.  Once added and balanced, we will complete the
copies (<3 days) and then migrate one more host allowing us to bring size
to 3.  Once balanced, we will collapse the last 2 nodes into the new
cluster.  I am hoping that inclusive of rebalancing the whole project will
only take 3 weeks, but time will tell.

Has anyone asked Ceph to reduce hundreds of millions if not billions of
files from size 3 to size 2, and if so, were you successful?  I know it
*should* be able to do this, but sometimes theory and practice don't
perfectly overlap.

Thanks,
Marco

On Sat, Dec 11, 2021 at 4:37 AM Martin Verges 
wrote:

> Hello,
>
> avoid size 2 whenever you can. As long as you know that you might lose
> data, it can be an acceptable risk while migrating the cluster. We had that
> in the past multiple time and it is a valid use case in our opinion.
> However make sure to monitor the state and recover as fast as possible.
> Leave min_size on 2 as well and accept the potential downtime!
>
> --
> Martin Verges
> Managing director
>
> Mobile: +49 174 9335695  | Chat: https://t.me/MartinVerges
>
> croit GmbH, Freseniusstr. 31h, 81247 Munich
> CEO: Martin Verges - VAT-ID: DE310638492
> Com. register: Amtsgericht Munich HRB 231263
> Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
>
>
> On Fri, 10 Dec 2021 at 18:05, Marco Pizzolo 
> wrote:
>
>> Hello,
>>
>> As part of a migration process where we will be swinging Ceph hosts from
>> one cluster to another we need to reduce the size from 3 to 2 in order to
>> shrink the footprint sufficiently to allow safe removal of an OSD/Mon
>> node.
>>
>> The cluster has about 500M objects as per dashboard, and is about 1.5PB in
>> size comprised solely of small files served through CephFS to Samba.
>>
>> Has anyone encountered a similar situation?  What (if any) problems did
>> you
>> face?
>>
>> Ceph 14.2.22 bare metal deployment on Centos.
>>
>> Thanks in advance.
>>
>> Marco
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Experience reducing size 3 to 2 on production cluster?

[Martin Verges]

Hello,

avoid size 2 whenever you can. As long as you know that you might lose
data, it can be an acceptable risk while migrating the cluster. We had that
in the past multiple time and it is a valid use case in our opinion.
However make sure to monitor the state and recover as fast as possible.
Leave min_size on 2 as well and accept the potential downtime!

--
Martin Verges
Managing director

Mobile: +49 174 9335695  | Chat: https://t.me/MartinVerges

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx


On Fri, 10 Dec 2021 at 18:05, Marco Pizzolo  wrote:

> Hello,
>
> As part of a migration process where we will be swinging Ceph hosts from
> one cluster to another we need to reduce the size from 3 to 2 in order to
> shrink the footprint sufficiently to allow safe removal of an OSD/Mon node.
>
> The cluster has about 500M objects as per dashboard, and is about 1.5PB in
> size comprised solely of small files served through CephFS to Samba.
>
> Has anyone encountered a similar situation?  What (if any) problems did you
> face?
>
> Ceph 14.2.22 bare metal deployment on Centos.
>
> Thanks in advance.
>
> Marco
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Experience reducing size 3 to 2 on production cluster?

[Wesley Dillingham]

I would avoid doing this. Size 2 is not where you want to be. Maybe you can
give more details about your cluster size and shape and what you are trying
to accomplish and another solution could be proposed. The contents of "ceph
osd tree " and "ceph df" would help.

Respectfully,

*Wes Dillingham*
w...@wesdillingham.com
LinkedIn 


On Fri, Dec 10, 2021 at 12:05 PM Marco Pizzolo 
wrote:

> Hello,
>
> As part of a migration process where we will be swinging Ceph hosts from
> one cluster to another we need to reduce the size from 3 to 2 in order to
> shrink the footprint sufficiently to allow safe removal of an OSD/Mon node.
>
> The cluster has about 500M objects as per dashboard, and is about 1.5PB in
> size comprised solely of small files served through CephFS to Samba.
>
> Has anyone encountered a similar situation?  What (if any) problems did you
> face?
>
> Ceph 14.2.22 bare metal deployment on Centos.
>
> Thanks in advance.
>
> Marco
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io