We have a separate primary and backup cluster running in two distinct physical locations serving rbd images (totaling ~12TB at the moment) to CIFS/NFS/iSCSI reshare hosts, serving clients . I do daily snapshots on the primary cluster and then export-diff/import-diff on the backup cluster, and then rotate the snapshots. This covers us if:
- Someone deletes data which has been picked up by the previous snapshot. I can mount the snapshot and I or the user can recover the affected file. - We run into something like J-P, where I get stuck and lose data on the primary cluster due to either a Ceph bug/error or mistake on my part. - We lose the primary site: flood/fire, etc. As a bonus, the backup cluster gives me a place to test upgrades/configuration changes before committing to them on the production system. I think Hammer allows for synchronous RBD mirroring between clusters (haven't played with that yet), and it looks like async mirroring is on the roadmap for Infernalis: https://wiki.ceph.com/Planning/Blueprints/Infernalis/RBD_Async_Mirroring -Steve On 05/06/2015 01:31 PM, J-P Methot wrote: > Case in point, here's a little story as to why backup outside ceph is > necessary: > > I was working on modifying journal locations for a running test ceph > cluster when, after bringing back a few OSD nodes, two PGs started > being marked as incomplete. That made all operations on the pool hang > as, for some reason, rbd clients couldn't read the missing PG and > there was no timeout value for their operation. After spending half a > day fixing this, I ended up needing to delete the pool and then > recreate it. Thankfully that setup was not in production so it was > only a minor setback. > > So, when we go in production with our setup, we are planning to have a > second ceph for backups, just in case such an issue happens again. I > don't want to scare anyone and I'm pretty sure my issue was very > exceptional, but no matter how well ceph replicate and ensures data > safety, backups are still a good idea, in my humble opinion. > > > On 5/6/2015 6:35 AM, Mariusz Gronczewski wrote: >> Snapshot on same storage cluster should definitely NOT be treated as >> backup >> >> Snapshot as a source for backup however can be pretty good solution for >> some cases, but not every case. >> >> For example if using ceph to serve static web files, I'd rather have >> possibility to restore given file from given path than snapshot of >> whole multiple TB cluster. >> >> There are 2 cases for backup restore: >> >> * something failed, need to fix it - usually full restore needed >> * someone accidentally removed a thing, and now they need a thing back >> >> Snapshots fix first problem, but not the second one, restoring 7TB of >> data to recover few GBs is not reasonable. >> >> As it is now we just backup from inside VMs (file-based backup) and have >> puppet to easily recreate machine config but if (or rather when) we >> would use object store we would backup it in a way that allows for >> partial restore. >> >> On Wed, 6 May 2015 10:50:34 +0100, Nick Fisk <[email protected]> wrote: >>> For me personally I would always feel more comfortable with backups on a >>> completely different storage technology. >>> >>> Whilst there are many things you can do with snapshots and replication, >>> there is always a small risk that whatever causes data loss on your primary >>> system may affect/replicate to your 2nd copy. >>> >>> I guess it all really depends on what you are trying to protect against, >>> but Tape still looks very appealing if you want to maintain a completely >>> isolated copy of data. >>> >>>> -----Original Message----- >>>> From: ceph-users [mailto:[email protected]] On Behalf Of >>>> Alexandre DERUMIER >>>> Sent: 06 May 2015 10:10 >>>> To: Götz Reinicke >>>> Cc: ceph-users >>>> Subject: Re: [ceph-users] How to backup hundreds or thousands of TB >>>> >>>> for the moment, you can use snapshot for backup >>>> >>>> https://ceph.com/community/blog/tag/backup/ >>>> >>>> I think that async mirror is on the roadmap >>>> https://wiki.ceph.com/Planning/Blueprints/Hammer/RBD%3A_Mirroring >>>> >>>> >>>> >>>> if you use qemu, you can do qemu full backup. (qemu incremental backup is >>>> coming for qemu 2.4) >>>> >>>> >>>> ----- Mail original ----- >>>> De: "Götz Reinicke" <[email protected]> >>>> À: "ceph-users" <[email protected]> >>>> Envoyé: Mercredi 6 Mai 2015 10:25:01 >>>> Objet: [ceph-users] How to backup hundreds or thousands of TB >>>> >>>> Hi folks, >>>> >>>> beside hardware and performance and failover design: How do you manage >>>> to backup hundreds or thousands of TB :) ? >>>> >>>> Any suggestions? Best practice? >>>> >>>> A second ceph cluster at a different location? "bigger archive" Disks in >>>> good >>>> boxes? Or tabe-libs? >>>> >>>> What kind of backupsoftware can handle such volumes nicely? >>>> >>>> Thanks and regards . Götz >>>> -- >>>> Götz Reinicke >>>> IT-Koordinator >>>> >>>> Tel. +49 7141 969 82 420 >>>> E-Mail [email protected] >>>> >>>> Filmakademie Baden-Württemberg GmbH >>>> Akademiehof 10 >>>> 71638 Ludwigsburg >>>> www.filmakademie.de >>>> >>>> Eintragung Amtsgericht Stuttgart HRB 205016 >>>> >>>> Vorsitzender des Aufsichtsrats: Jürgen Walter MdL Staatssekretär im >>>> Ministerium für Wissenschaft, Forschung und Kunst Baden-Württemberg >>>> >>>> Geschäftsführer: Prof. Thomas Schadt >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> [email protected] >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> _______________________________________________ >>>> ceph-users mailing list >>>> [email protected] >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> [email protected] >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> _______________________________________________ >> ceph-users mailing list >> [email protected] >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > -- > ====================== > Jean-Philippe Méthot > Administrateur système / System administrator > GloboTech Communications > Phone: 1-514-907-0050 > Toll Free: 1-(888)-GTCOMM1 > Fax: 1-(514)-907-0750 > [email protected] > http://www.gtcomm.net > > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Steve Anthony LTS HPC Support Specialist Lehigh University [email protected]
signature.asc
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
