On 7/24/19 4:06 PM, Fabian Niepelt wrote:
> Hi, thanks for the reply.
>
> Am Mittwoch, den 24.07.2019, 15:26 +0200 schrieb Wido den Hollander:
>>
>> On 7/24/19 1:37 PM, Fabian Niepelt wrote:
>>> Hello ceph-users,
>>>
>>> I am currently building a Ceph cluster that will serve as a backend for
>>> Openstack and object storage using RGW. The cluster itself is finished and
>>> integrated with Openstack and virtual machines for testing are being
>>> deployed.
>>> Now I'm a bit stumped on how to effectively backup the Ceph pools.
>>> My requirements are two weekly backups, of which one must be offline after
>>> finishing backing up (systems turned powerless). We are expecting about
>>> 250TB to
>>> 500TB of data for now. The backups must protect against accidental pool
>>> deletion/corruption or widespread infection of a cryptovirus. In short:
>>> Complete
>>> data loss in the production Ceph cluster.
>>>
>>> At the moment, I am facing two issues:
>>>
>>> 1. For the cinder pool, I looked into creating snapshots using the ceph CLI
>>> (so
>>> they don't turn up in Openstack and cannot be accidentally deleted by users)
>>> and
>>> exporting their diffs. But volumes with snapshots created this way cannot be
>>> removed from Openstack. Does anyone have an idea how to do this better?
>>
>> You mean that while you leave the snapshot there OpenStack can't remove it?
>
> Yes, that is correct. cinder-volume cannot remove a volume that still has a
> snapshot. If the snapshot is created by openstack, it will remove the snapshot
> before removing the volume. But snapshotting directly from ceph will forego
> Openstack so it will never know about that snapshot's existence.
>
Ah, yes. That means you would need to remove it manually.
>>> Alternatively, I could do a full export each week, but I am not sure if that
>>> would be fast enough..
>>>
>>
>> It probably won't, but the full backup is still the safest way imho.
>> However: Does this scale?
>>
>> You can export multiple RBD images in parallel and store them somewhere
>> else, but it will still take a long time.
>>
>> The export needs to be stored somewhere and then picked up. Or you could
>> use some magic with Netcat to stream the RBD export to a destination host.
>>
>
> Scaling is also my biggest worry about this.
>
>>> 2. My search so far has only turned up backing up RBD pools, but how could I
>>> backup the pools that are used for object storage?
>>>
>>
>> Not easily. I think you mean RGW? You could try the RGW MultiSite, but
>> it's difficult.
>>
>> A complete DR with Ceph to restore it back to how it was at a given
>> point in time is a challenge.
>>
>
> Yes, I would like to backup the pools used by the RGW.
Not really an option. You would need to use the RGW MultiSite to
replicate all data to a second environment.
>
>>> Of course, I'm also open to completely other ideas on how to backup Ceph and
>>> would appreciate hearing how you people are doing your backups.
>>
>> A lot of time the backups are created inside the VMs on File level. And
>> there is a second OpenStack+Ceph system which runs a mirror of the VMs
>> or application. If one burns down it's not the end of the world.
>>
>> Trying to backup a Ceph cluster sounds very 'enterprise' and is
>> difficult to scale as well.
>>
>
> Are those backups saved in Ceph as well? I cannot solely rely on Ceph because
> we
> want to protect ourselves against failures in Ceph or a human accidentally or
> maliciously deletes all pools.
>
That second OpenStack+Ceph environment is completely different. All the
VMs are set up twice and using replication and backups on application
level such things are redundant.
Think about MySQL replication for example.
> From what I'm reading, it seems to be better to maybe implement a backup
> solution outside of Ceph that our Openstack users can use and not deal with
> backing up Ceph at all, except its configs to get it running after total
> desaster...
>
You could backup OpenStack's MySQL database, the ceph config and then
backup the data inside the VMs.
It's very difficult to backup data for DR to a certain point of time
when you go into the >100TB scale.
Wido
>> Wido
>>
>>> Any help is much appreciated.
>>>
>>> Greetings
>>> Fabian
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
