Thanks for the details Nicolas. I do think the picture is clear(er?) now. At least, now I understand what you are doing.
BTW: It is a very interesting feature ;) On Thu, Jun 7, 2018 at 8:30 PM, Will Stevens <wstev...@cloudops.com> wrote: > Thanks for clarifying Nicolas. I just wanted to make sure that there > wasn't any confusion in this conversation as a 'snapshot' has different > meanings to different people/vendors, so I wanted to make sure we were all > on the same page. > > BTW, for what it is worth, I love this initiative. This is an important > feature integration which I know a LOT of organizations will appreciate. > > Cheers, > > *Will Stevens* > Chief Technology Officer > c 514.826.0190 > > <https://goo.gl/NYZ8KK> > > > On Thu, Jun 7, 2018 at 12:57 PM Nicolas Vazquez < > nicolas.vazq...@shapeblue.com> wrote: > > > First of all, the framework is agnostic of any backup vendor/provider. It > > just provides the way to communicate with the provider to allow users to > > backup their VMs. > > > > > > Will is right, backups are not the same as snapshots, I got confused with > > the Veeam backup which starts from taking a snapshot to generate the > > backup. The way they are generated are up to each vendor, and it is not > > relevant to the framework. Sorry for the confusion, we are also > introducing > > the Veeam plugin in top of this framework. > > > > > > So, in terms of this framework, it does not matter what a backup is, we > > just keep a reference to an existing backup on each vendor. > > > > > > I would also like to state that this framework is not using a snapshot > > mechanism for backups in CloudStack. It is just providing interfaces and > > mechanisms for vendors/providers plugins in CloudStack. Implementation of > > each plugin should be relevant to the vendor and any backup handling will > > be external to CloudStack. > > > > > > I think that the diagram in the FS could explain what the task of the > > framework is: > > https://cwiki.apache.org/confluence/download/ > attachments/85476500/Backup%20and%20Recovery%20UML.png? > version=1&modificationDate=1528367734183&api=v2. > > The framework simply allows management server to communicate with backup > > and recovery vendors/providers. To allow CloudStack communication with an > > specific vendor, a plugin should be implemented for it (as we are doing > > with Veeam for example). > > > > > > Please let me know if it is still confusing and I'll try to explain > better > > > > ________________________________ > > From: Todd Pigram <t...@toddpigram.com> > > Sent: Thursday, June 7, 2018 12:11:18 PM > > To: dev@cloudstack.apache.org > > Subject: Re: [PROPOSE] Backup and Recovery Framework > > > > Now outside of our ACS, I use Veeam for vCloud and vSphere and use Alike > > for our XS pools. Inside ACS, the VMs use MaxBackup > > > > > > nicolas.vazq...@shapeblue.com > > www.shapeblue.com > > , > > @shapeblue > > > > > > > > On Thu, Jun 7, 2018 at 11:09 AM, Todd Pigram <t...@toddpigram.com> > wrote: > > > > > I agree completely. That is why I use inside VM backup products > > currently. > > > > > > On Thu, Jun 7, 2018 at 11:07 AM, Will Stevens <wstev...@cloudops.com> > > > wrote: > > > > > >> I have not been following this thread entirely, but I would not count > on > > >> our snapshot mechanism for backups. If something like Veeam is used, > > then > > >> an agent is in play and that agent will quiesce the running processes, > > >> such > > >> as MySQL. This is important in order to do backup and DR as you will > > lose > > >> transactions in memory and can potentially have a corrupted backup. > > >> > > >> Snapshots are not backups!!! > > >> > > >> *Will Stevens* > > >> Chief Technology Officer > > >> c 514.826.0190 > > >> > > >> <https://goo.gl/NYZ8KK> > > >> > > >> > > >> On Thu, Jun 7, 2018 at 10:13 AM Rafael Weingärtner < > > >> rafaelweingart...@gmail.com> wrote: > > >> > > >> > Ok, now I start to understand. So, we will be handling snapshots > and > > >> they > > >> > will be sent to a third-party system. > > >> > > > >> > Now let’s proceed. If I configure this “feature” with > external=false. > > >> What > > >> > does it do? Nothing? I mean, what is the purpose of this “external” > > >> > parameter? > > >> > > > >> > > > >> > On Thu, Jun 7, 2018 at 4:09 PM, Nicolas Vazquez < > > >> > nicolas.vazq...@shapeblue.com> wrote: > > >> > > > >> > > Its format may depend on the provider, but under the hood it is a > > >> > snapshot > > >> > > which can be full or incremental and it is copied and stored in a > > >> backup > > >> > > repository in one of the backup servers. For example, if you use > > >> Veeam, > > >> > you > > >> > > will find your backups stored as .vbk (for full backups) and .vib > > (for > > >> > > incremental backups). It is the task of the backup provider to > > create > > >> and > > >> > > move these files to the backup repository. > > >> > > > > >> > > ________________________________ > > >> > > From: Rafael Weingärtner <rafaelweingart...@gmail.com> > > >> > > Sent: Thursday, June 7, 2018 10:39:35 AM > > >> > > To: dev > > >> > > Subject: Re: [PROPOSE] Backup and Recovery Framework > > >> > > > > >> > > Ok, there are a lot of things, which are kind of blurry to me. So, > > >> let’s > > >> > > try to go step by step. > > >> > > > > >> > > Let me ask again. What is a backup of the VM? You mention > > >> restore-point. > > >> > > What is it? Is it a snapshot in the hypervisor (complete/partial > > >> > snapshot)? > > >> > > If it is not a snapshot, what is it then? Are you installing some > > >> agent > > >> > in > > >> > > the VM and then copying/mirroring the files in the VM’s file > system? > > >> > > > > >> > > On Thu, Jun 7, 2018 at 3:26 PM, Nicolas Vazquez < > > >> > > nicolas.vazq...@shapeblue.com> wrote: > > >> > > > > >> > > > Hi Daniel and Rafael, > > >> > > > > > >> > > > > > >> > > > I have updated the diagram picture on the FS, please let me know > > if > > >> the > > >> > > > problem persists. > > >> > > > > > >> > > > > > >> > > > @<mailto:daniel.herrm...@zv.fraunhofer.de>Daniel you are right > > >> about > > >> > > > restoring VMs, please let me update the FS. Regarding your > > >> question, we > > >> > > > have considered the backups as a first class citizens, meaning > > that > > >> a > > >> > > > backup would continue existing unless its originating VM has > been > > >> > > > removed/expunged. This way, listBackups API method would list > them > > >> in > > >> > > order > > >> > > > to allow a user to restore a deleted VM from a backup. > > >> > > > > > >> > > > > > >> > > > @Rafael the idea of the Dummy provider is simply returning the > > >> expected > > >> > > > entities from a backup provider. For example, the BackupManager > > >> could > > >> > ask > > >> > > > the BackupProvider to list its backups and would expect a list > of > > >> > > backups. > > >> > > > As the Dummy Provider implements the BackupProvider interface, > it > > >> has > > >> > its > > >> > > > own implementation of listBackups() method and simply returns > some > > >> mock > > >> > > > entities to the Backup Manager. It does not store any backup, it > > >> simply > > >> > > > creates them in execution time and returns them. > > >> > > > > > >> > > > > > >> > > > From the CloudStack point of view, Backups are stored > externally, > > on > > >> > the > > >> > > > Backup and Recovery provider servers and are handled by them. We > > >> keep > > >> > > track > > >> > > > of the existing backups in the Backup provider. Implementation > may > > >> > depend > > >> > > > on the Backup and Recovery provider, would consist on snapshots, > > but > > >> > are > > >> > > > handled by the provider and not by CloudStack. > > >> > > > > > >> > > > > > >> > > > Backup policies are created on the Backup and Recovery provider > > and > > >> > > > provide retention options and periods as well as backup times. > For > > >> > > example, > > >> > > > the provider may expose Golden, Silver and Bronze policies and > > have > > >> > > > different options for each of them. > > >> > > > > > >> > > > > > >> > > > The 'external' parameter on the listBackupPolicies API method > > would > > >> > only > > >> > > > ask the Backup and Recovery provider to list their policies if > it > > is > > >> > set > > >> > > to > > >> > > > true. If it is not set, or set to false, it just simply returns > > the > > >> > > > existing mapped policies in CloudStack. Initially, CloudStack > does > > >> not > > >> > > > store any backup policy, those should be imported/mapped by the > > >> > > > 'importBackupPolicy' API method. > > >> > > > > > >> > > > > > >> > > > Finally, we consider backups as restore points. As mentioned > > above, > > >> > > > backups are stored on the Backup and Recovery provider servers, > so > > >> we > > >> > > keep > > >> > > > a mapping between the CloudStack entity and the backup, which > can > > be > > >> > used > > >> > > > to restore VMs. > > >> > > > > > >> > > > > > >> > > > Please let me know if anything. > > >> > > > > > >> > > > > > >> > > > Regards, > > >> > > > > > >> > > > Nicolas Vazquez > > >> > > > > > >> > > > ________________________________ > > >> > > > From: Rafael Weingärtner <rafaelweingart...@gmail.com> > > >> > > > Sent: Thursday, June 7, 2018 6:55:57 AM > > >> > > > To: dev > > >> > > > Subject: Re: [PROPOSE] Backup and Recovery Framework > > >> > > > > > >> > > > Hi Nicolas, > > >> > > > > > >> > > > The idea is great. I have some questions though. > > >> > > > > > >> > > > Before getting to the questions, your pictures are broken, or at > > >> least > > >> > > they > > >> > > > seem broken to me. > > >> > > > > > >> > > > How would the “dummy” backup provider work? I mean, what does it > > do? > > >> > > where > > >> > > > does it storage the "backup"? > > >> > > > What is the difference from this Backup to a Snapshot? Is it > > simply > > >> > > storing > > >> > > > a snapshot outside the cloud structure? > > >> > > > How would these “backup policies” look like? I mean, what can I > > >> > > configure? > > >> > > > Regarding that parameter “external” of your backup policy, how > > >> would it > > >> > > > work if we set it to false? > > >> > > > What are you defining as a backup (what do you mean by backup in > > the > > >> > > > context of this feature?)? I mean, is it simply a snapshot file > > that > > >> > you > > >> > > > store somewhere else? > > >> > > > > > >> > > > > > >> > > > nicolas.vazq...@shapeblue.com > > >> > > > www.shapeblue.com<http://www.shapeblue.com> > > >> > > > , > > >> > > > @shapeblue > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > nicolas.vazq...@shapeblue.com > > >> > > www.shapeblue.com<http://www.shapeblue.com> > > >> > > , > > >> > > @shapeblue > > >> > > > > >> > > > > >> > > > > >> > > > On Thu, Jun 7, 2018 at 6:05 AM, <daniel.herrmann@zv. > fraunhofer.de > > > > > >> > > wrote: > > >> > > > > > >> > > > > Hi all, > > >> > > > > > > >> > > > > +1 for this feature request, thank you for bringing this to > > life! > > >> > > Volume > > >> > > > > snapshots are what our users currently use as backup solution, > > >> which > > >> > > has > > >> > > > > some inherent problems such as handling large volumes and the > > >> > handling > > >> > > of > > >> > > > > multiple volumes (consistency). > > >> > > > > > > >> > > > > One thought: when restoring a VM, I think there are two cases > to > > >> > > > consider: > > >> > > > > > > >> > > > > - The VM is still in CS and the user just wants to restore an > > old > > >> > > version > > >> > > > > of the VM > > >> > > > > - The VM has already been deleted and the user wants to > restore > > >> the > > >> > > > > deleted VM > > >> > > > > > > >> > > > > As listBackups is described as " listBackups API method. List > > >> > existing > > >> > > > > backups for a VM". Is the second use case supported at all? > > >> > > > > > > >> > > > > I like the approach not to define the backup policies in CS > but > > in > > >> > > > > whatever tool is used in the end, as otherwise a lot of > features > > >> > might > > >> > > be > > >> > > > > lost. We as services providers need to create them in the > first > > >> > place, > > >> > > > but > > >> > > > > I think this is the better solution. > > >> > > > > > > >> > > > > Regards > > >> > > > > Daniel > > >> > > > > > > >> > > > > On 06.06.18, 14:43, "Nicolas Vazquez" < > > >> > nicolas.vazq...@shapeblue.com> > > >> > > > > wrote: > > >> > > > > > > >> > > > > Hi all, > > >> > > > > > > >> > > > > > > >> > > > > We would like to introduce a new framework into CloudStack > > >> that > > >> > > will > > >> > > > > allow adding Backup and Recovery providers as plugins. This > > >> framework > > >> > > > goal > > >> > > > > is allowing users to backup their guest VMs for recovery > > purposes. > > >> > > > > > > >> > > > > > > >> > > > > Please find the FS on this link: > https://cwiki.apache.org/ > > >> > > > > confluence/display/CLOUDSTACK/Backup+and+Recovery+Framework > > >> > > > > > > >> > > > > > > >> > > > > Regards, > > >> > > > > > > >> > > > > Nicolas Vazquez > > >> > > > > > > >> > > > > nicolas.vazq...@shapeblue.com > > >> > > > > www.shapeblue.com<http://www.shapeblue.com> > > >> > > > > , > > >> > > > > @shapeblue > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > -- > > >> > > > Rafael Weingärtner > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > -- > > >> > > Rafael Weingärtner > > >> > > > > >> > > > >> > > > >> > > > >> > -- > > >> > Rafael Weingärtner > > >> > > > >> > > > > > > > > > > > > -- > > > > > > > > > Todd Pigram > > > http://about.me/ToddPigram > > > www.linkedin.com/in/toddpigram/<http://www.linkedin.com/in/toddpigram/ > > > > > @pigram86 on twitter > > > https://plus.google.com/+ToddPigram86 > > > Mobile - 216-224-5769 > > > > > > PGP Public Key > > > <http://pgp.mit.edu/pks/lookup?op=get&search=0x96B7B0F0C55933BB> > > > > > > > > > > > > -- > > > > > > Todd Pigram > > http://about.me/ToddPigram > > www.linkedin.com/in/toddpigram/<http://www.linkedin.com/in/toddpigram/> > > @pigram86 on twitter > > https://plus.google.com/+ToddPigram86 > > Mobile - 216-224-5769 > > > > PGP Public Key > > <http://pgp.mit.edu/pks/lookup?op=get&search=0x96B7B0F0C55933BB> > > > -- Rafael Weingärtner
