Re: [PROPOSE] Backup and Recovery Framework
Thanks guys. Yes, we are planning the integration with Veeam as a plugin within this framework. Kind regards, Nicolas De: Syed Ahmed Enviado: lunes, 11 de junio 11:19 Asunto: Re: [PROPOSE] Backup and Recovery Framework Para: dev@cloudstack.apache.org +1 for this feature Nicolas. As Will had mentioned, snapshots are not backups. snapshots just ensure crash consistency but don’t necessarily guarantee data consistency. My question is that whether there is going to be a 3rd party backup solution integration that will serve as a reference? Thanks, -Syed nicolas.vazq...@shapeblue.com www.shapeblue.com , @shapeblue On Mon, Jun 11, 2018 at 5:57 AM Rafael Weingärtner < rafaelweingart...@gmail.com> wrote: > Thanks for the details Nicolas. I do think the picture is clear(er?) now. > At least, now I understand what you are doing. > > BTW: It is a very interesting feature ;) > > On Thu, Jun 7, 2018 at 8:30 PM, Will Stevens > wrote: > > > Thanks for clarifying Nicolas. I just wanted to make sure that there > > wasn't any confusion in this conversation as a 'snapshot' has different > > meanings to different people/vendors, so I wanted to make sure we were > all > > on the same page. > > > > BTW, for what it is worth, I love this initiative. This is an important > > feature integration which I know a LOT of organizations will appreciate. > > > > Cheers, > > > > *Will Stevens* > > Chief Technology Officer > > c 514.826.0190 > > > > > > > > > > On Thu, Jun 7, 2018 at 12:57 PM Nicolas Vazquez < > > nicolas.vazq...@shapeblue.com> wrote: > > > > > First of all, the framework is agnostic of any backup vendor/provider. > It > > > just provides the way to communicate with the provider to allow users > to > > > backup their VMs. > > > > > > > > > Will is right, backups are not the same as snapshots, I got confused > with > > > the Veeam backup which starts from taking a snapshot to generate the > > > backup. The way they are generated are up to each vendor, and it is not > > > relevant to the framework. Sorry for the confusion, we are also > > introducing > > > the Veeam plugin in top of this framework. > > > > > > > > > So, in terms of this framework, it does not matter what a backup is, we > > > just keep a reference to an existing backup on each vendor. > > > > > > > > > I would also like to state that this framework is not using a snapshot > > > mechanism for backups in CloudStack. It is just providing interfaces > and > > > mechanisms for vendors/providers plugins in CloudStack. Implementation > of > > > each plugin should be relevant to the vendor and any backup handling > will > > > be external to CloudStack. > > > > > > > > > I think that the diagram in the FS could explain what the task of the > > > framework is: > > > https://cwiki.apache.org/confluence/download/ > > attachments/85476500/Backup%20and%20Recovery%20UML.png? > > version=1=1528367734183=v2. > > > The framework simply allows management server to communicate with > backup > > > and recovery vendors/providers. To allow CloudStack communication with > an > > > specific vendor, a plugin should be implemented for it (as we are doing > > > with Veeam for example). > > > > > > > > > Please let me know if it is still confusing and I'll try to explain > > better > > > > > > > > > From: Todd Pigram > > > Sent: Thursday, June 7, 2018 12:11:18 PM > > > To: dev@cloudstack.apache.org > > > Subject: Re: [PROPOSE] Backup and Recovery Framework > > > > > > Now outside of our ACS, I use Veeam for vCloud and vSphere and use > Alike > > > for our XS pools. Inside ACS, the VMs use MaxBackup > > > > > > > > > nicolas.vazq...@shapeblue.com > > > www.shapeblue.com > > > , > > > @shapeblue > > > > > > > > > > > > On Thu, Jun 7, 2018 at 11:09 AM, Todd Pigram > > wrote: > > > > > > > I agree completely. That is why I use inside VM backup products > > > currently. > > > > > > > > On Thu, Jun 7, 2018 at 11:07 AM, Will Stevens > > > > > wrote: > > > > > > > >> I have not been following this thread entirely, but I would not > count > > on > > > >> our snapshot mechanism for backups. If somet
Re: [PROPOSE] Backup and Recovery Framework
+1 for this feature Nicolas. As Will had mentioned, snapshots are not backups. snapshots just ensure crash consistency but don’t necessarily guarantee data consistency. My question is that whether there is going to be a 3rd party backup solution integration that will serve as a reference? Thanks, -Syed On Mon, Jun 11, 2018 at 5:57 AM Rafael Weingärtner < rafaelweingart...@gmail.com> wrote: > Thanks for the details Nicolas. I do think the picture is clear(er?) now. > At least, now I understand what you are doing. > > BTW: It is a very interesting feature ;) > > On Thu, Jun 7, 2018 at 8:30 PM, Will Stevens > wrote: > > > Thanks for clarifying Nicolas. I just wanted to make sure that there > > wasn't any confusion in this conversation as a 'snapshot' has different > > meanings to different people/vendors, so I wanted to make sure we were > all > > on the same page. > > > > BTW, for what it is worth, I love this initiative. This is an important > > feature integration which I know a LOT of organizations will appreciate. > > > > Cheers, > > > > *Will Stevens* > > Chief Technology Officer > > c 514.826.0190 > > > > <https://goo.gl/NYZ8KK> > > > > > > On Thu, Jun 7, 2018 at 12:57 PM Nicolas Vazquez < > > nicolas.vazq...@shapeblue.com> wrote: > > > > > First of all, the framework is agnostic of any backup vendor/provider. > It > > > just provides the way to communicate with the provider to allow users > to > > > backup their VMs. > > > > > > > > > Will is right, backups are not the same as snapshots, I got confused > with > > > the Veeam backup which starts from taking a snapshot to generate the > > > backup. The way they are generated are up to each vendor, and it is not > > > relevant to the framework. Sorry for the confusion, we are also > > introducing > > > the Veeam plugin in top of this framework. > > > > > > > > > So, in terms of this framework, it does not matter what a backup is, we > > > just keep a reference to an existing backup on each vendor. > > > > > > > > > I would also like to state that this framework is not using a snapshot > > > mechanism for backups in CloudStack. It is just providing interfaces > and > > > mechanisms for vendors/providers plugins in CloudStack. Implementation > of > > > each plugin should be relevant to the vendor and any backup handling > will > > > be external to CloudStack. > > > > > > > > > I think that the diagram in the FS could explain what the task of the > > > framework is: > > > https://cwiki.apache.org/confluence/download/ > > attachments/85476500/Backup%20and%20Recovery%20UML.png? > > version=1=1528367734183=v2. > > > The framework simply allows management server to communicate with > backup > > > and recovery vendors/providers. To allow CloudStack communication with > an > > > specific vendor, a plugin should be implemented for it (as we are doing > > > with Veeam for example). > > > > > > > > > Please let me know if it is still confusing and I'll try to explain > > better > > > > > > > > > From: Todd Pigram > > > Sent: Thursday, June 7, 2018 12:11:18 PM > > > To: dev@cloudstack.apache.org > > > Subject: Re: [PROPOSE] Backup and Recovery Framework > > > > > > Now outside of our ACS, I use Veeam for vCloud and vSphere and use > Alike > > > for our XS pools. Inside ACS, the VMs use MaxBackup > > > > > > > > > nicolas.vazq...@shapeblue.com > > > www.shapeblue.com > > > , > > > @shapeblue > > > > > > > > > > > > On Thu, Jun 7, 2018 at 11:09 AM, Todd Pigram > > wrote: > > > > > > > I agree completely. That is why I use inside VM backup products > > > currently. > > > > > > > > On Thu, Jun 7, 2018 at 11:07 AM, Will Stevens > > > > > wrote: > > > > > > > >> I have not been following this thread entirely, but I would not > count > > on > > > >> our snapshot mechanism for backups. If something like Veeam is > used, > > > then > > > >> an agent is in play and that agent will quiesce the running > processes, > > > >> such > > > >> as MySQL. This is important in order to do backup and DR as you > will > > > lose > > > >> transactions in memory
Re: [PROPOSE] Backup and Recovery Framework
Thanks for the details Nicolas. I do think the picture is clear(er?) now. At least, now I understand what you are doing. BTW: It is a very interesting feature ;) On Thu, Jun 7, 2018 at 8:30 PM, Will Stevens wrote: > Thanks for clarifying Nicolas. I just wanted to make sure that there > wasn't any confusion in this conversation as a 'snapshot' has different > meanings to different people/vendors, so I wanted to make sure we were all > on the same page. > > BTW, for what it is worth, I love this initiative. This is an important > feature integration which I know a LOT of organizations will appreciate. > > Cheers, > > *Will Stevens* > Chief Technology Officer > c 514.826.0190 > > <https://goo.gl/NYZ8KK> > > > On Thu, Jun 7, 2018 at 12:57 PM Nicolas Vazquez < > nicolas.vazq...@shapeblue.com> wrote: > > > First of all, the framework is agnostic of any backup vendor/provider. It > > just provides the way to communicate with the provider to allow users to > > backup their VMs. > > > > > > Will is right, backups are not the same as snapshots, I got confused with > > the Veeam backup which starts from taking a snapshot to generate the > > backup. The way they are generated are up to each vendor, and it is not > > relevant to the framework. Sorry for the confusion, we are also > introducing > > the Veeam plugin in top of this framework. > > > > > > So, in terms of this framework, it does not matter what a backup is, we > > just keep a reference to an existing backup on each vendor. > > > > > > I would also like to state that this framework is not using a snapshot > > mechanism for backups in CloudStack. It is just providing interfaces and > > mechanisms for vendors/providers plugins in CloudStack. Implementation of > > each plugin should be relevant to the vendor and any backup handling will > > be external to CloudStack. > > > > > > I think that the diagram in the FS could explain what the task of the > > framework is: > > https://cwiki.apache.org/confluence/download/ > attachments/85476500/Backup%20and%20Recovery%20UML.png? > version=1=1528367734183=v2. > > The framework simply allows management server to communicate with backup > > and recovery vendors/providers. To allow CloudStack communication with an > > specific vendor, a plugin should be implemented for it (as we are doing > > with Veeam for example). > > > > > > Please let me know if it is still confusing and I'll try to explain > better > > > > > > From: Todd Pigram > > Sent: Thursday, June 7, 2018 12:11:18 PM > > To: dev@cloudstack.apache.org > > Subject: Re: [PROPOSE] Backup and Recovery Framework > > > > Now outside of our ACS, I use Veeam for vCloud and vSphere and use Alike > > for our XS pools. Inside ACS, the VMs use MaxBackup > > > > > > nicolas.vazq...@shapeblue.com > > www.shapeblue.com > > , > > @shapeblue > > > > > > > > On Thu, Jun 7, 2018 at 11:09 AM, Todd Pigram > wrote: > > > > > I agree completely. That is why I use inside VM backup products > > currently. > > > > > > On Thu, Jun 7, 2018 at 11:07 AM, Will Stevens > > > wrote: > > > > > >> I have not been following this thread entirely, but I would not count > on > > >> our snapshot mechanism for backups. If something like Veeam is used, > > then > > >> an agent is in play and that agent will quiesce the running processes, > > >> such > > >> as MySQL. This is important in order to do backup and DR as you will > > lose > > >> transactions in memory and can potentially have a corrupted backup. > > >> > > >> Snapshots are not backups!!! > > >> > > >> *Will Stevens* > > >> Chief Technology Officer > > >> c 514.826.0190 > > >> > > >> <https://goo.gl/NYZ8KK> > > >> > > >> > > >> On Thu, Jun 7, 2018 at 10:13 AM Rafael Weingärtner < > > >> rafaelweingart...@gmail.com> wrote: > > >> > > >> > Ok, now I start to understand. So, we will be handling snapshots > and > > >> they > > >> > will be sent to a third-party system. > > >> > > > >> > Now let’s proceed. If I configure this “feature” with > external=false. > > >> What > > >> > does it do? Nothing? I mean, what is the purpose of this “external” > > >> > parameter? > > >> >
Re: [PROPOSE] Backup and Recovery Framework
Thanks for clarifying Nicolas. I just wanted to make sure that there wasn't any confusion in this conversation as a 'snapshot' has different meanings to different people/vendors, so I wanted to make sure we were all on the same page. BTW, for what it is worth, I love this initiative. This is an important feature integration which I know a LOT of organizations will appreciate. Cheers, *Will Stevens* Chief Technology Officer c 514.826.0190 <https://goo.gl/NYZ8KK> On Thu, Jun 7, 2018 at 12:57 PM Nicolas Vazquez < nicolas.vazq...@shapeblue.com> wrote: > First of all, the framework is agnostic of any backup vendor/provider. It > just provides the way to communicate with the provider to allow users to > backup their VMs. > > > Will is right, backups are not the same as snapshots, I got confused with > the Veeam backup which starts from taking a snapshot to generate the > backup. The way they are generated are up to each vendor, and it is not > relevant to the framework. Sorry for the confusion, we are also introducing > the Veeam plugin in top of this framework. > > > So, in terms of this framework, it does not matter what a backup is, we > just keep a reference to an existing backup on each vendor. > > > I would also like to state that this framework is not using a snapshot > mechanism for backups in CloudStack. It is just providing interfaces and > mechanisms for vendors/providers plugins in CloudStack. Implementation of > each plugin should be relevant to the vendor and any backup handling will > be external to CloudStack. > > > I think that the diagram in the FS could explain what the task of the > framework is: > https://cwiki.apache.org/confluence/download/attachments/85476500/Backup%20and%20Recovery%20UML.png?version=1=1528367734183=v2. > The framework simply allows management server to communicate with backup > and recovery vendors/providers. To allow CloudStack communication with an > specific vendor, a plugin should be implemented for it (as we are doing > with Veeam for example). > > > Please let me know if it is still confusing and I'll try to explain better > > > From: Todd Pigram > Sent: Thursday, June 7, 2018 12:11:18 PM > To: dev@cloudstack.apache.org > Subject: Re: [PROPOSE] Backup and Recovery Framework > > Now outside of our ACS, I use Veeam for vCloud and vSphere and use Alike > for our XS pools. Inside ACS, the VMs use MaxBackup > > > nicolas.vazq...@shapeblue.com > www.shapeblue.com > , > @shapeblue > > > > On Thu, Jun 7, 2018 at 11:09 AM, Todd Pigram wrote: > > > I agree completely. That is why I use inside VM backup products > currently. > > > > On Thu, Jun 7, 2018 at 11:07 AM, Will Stevens > > wrote: > > > >> I have not been following this thread entirely, but I would not count on > >> our snapshot mechanism for backups. If something like Veeam is used, > then > >> an agent is in play and that agent will quiesce the running processes, > >> such > >> as MySQL. This is important in order to do backup and DR as you will > lose > >> transactions in memory and can potentially have a corrupted backup. > >> > >> Snapshots are not backups!!! > >> > >> *Will Stevens* > >> Chief Technology Officer > >> c 514.826.0190 > >> > >> <https://goo.gl/NYZ8KK> > >> > >> > >> On Thu, Jun 7, 2018 at 10:13 AM Rafael Weingärtner < > >> rafaelweingart...@gmail.com> wrote: > >> > >> > Ok, now I start to understand. So, we will be handling snapshots and > >> they > >> > will be sent to a third-party system. > >> > > >> > Now let’s proceed. If I configure this “feature” with external=false. > >> What > >> > does it do? Nothing? I mean, what is the purpose of this “external” > >> > parameter? > >> > > >> > > >> > On Thu, Jun 7, 2018 at 4:09 PM, Nicolas Vazquez < > >> > nicolas.vazq...@shapeblue.com> wrote: > >> > > >> > > Its format may depend on the provider, but under the hood it is a > >> > snapshot > >> > > which can be full or incremental and it is copied and stored in a > >> backup > >> > > repository in one of the backup servers. For example, if you use > >> Veeam, > >> > you > >> > > will find your backups stored as .vbk (for full backups) and .vib > (for > >> > > incremental backups). It is the task of the backup provider to > create > >> and > >> > > move these files t
Re: [PROPOSE] Backup and Recovery Framework
First of all, the framework is agnostic of any backup vendor/provider. It just provides the way to communicate with the provider to allow users to backup their VMs. Will is right, backups are not the same as snapshots, I got confused with the Veeam backup which starts from taking a snapshot to generate the backup. The way they are generated are up to each vendor, and it is not relevant to the framework. Sorry for the confusion, we are also introducing the Veeam plugin in top of this framework. So, in terms of this framework, it does not matter what a backup is, we just keep a reference to an existing backup on each vendor. I would also like to state that this framework is not using a snapshot mechanism for backups in CloudStack. It is just providing interfaces and mechanisms for vendors/providers plugins in CloudStack. Implementation of each plugin should be relevant to the vendor and any backup handling will be external to CloudStack. I think that the diagram in the FS could explain what the task of the framework is: https://cwiki.apache.org/confluence/download/attachments/85476500/Backup%20and%20Recovery%20UML.png?version=1=1528367734183=v2. The framework simply allows management server to communicate with backup and recovery vendors/providers. To allow CloudStack communication with an specific vendor, a plugin should be implemented for it (as we are doing with Veeam for example). Please let me know if it is still confusing and I'll try to explain better From: Todd Pigram Sent: Thursday, June 7, 2018 12:11:18 PM To: dev@cloudstack.apache.org Subject: Re: [PROPOSE] Backup and Recovery Framework Now outside of our ACS, I use Veeam for vCloud and vSphere and use Alike for our XS pools. Inside ACS, the VMs use MaxBackup nicolas.vazq...@shapeblue.com www.shapeblue.com , @shapeblue On Thu, Jun 7, 2018 at 11:09 AM, Todd Pigram wrote: > I agree completely. That is why I use inside VM backup products currently. > > On Thu, Jun 7, 2018 at 11:07 AM, Will Stevens > wrote: > >> I have not been following this thread entirely, but I would not count on >> our snapshot mechanism for backups. If something like Veeam is used, then >> an agent is in play and that agent will quiesce the running processes, >> such >> as MySQL. This is important in order to do backup and DR as you will lose >> transactions in memory and can potentially have a corrupted backup. >> >> Snapshots are not backups!!! >> >> *Will Stevens* >> Chief Technology Officer >> c 514.826.0190 >> >> <https://goo.gl/NYZ8KK> >> >> >> On Thu, Jun 7, 2018 at 10:13 AM Rafael Weingärtner < >> rafaelweingart...@gmail.com> wrote: >> >> > Ok, now I start to understand. So, we will be handling snapshots and >> they >> > will be sent to a third-party system. >> > >> > Now let’s proceed. If I configure this “feature” with external=false. >> What >> > does it do? Nothing? I mean, what is the purpose of this “external” >> > parameter? >> > >> > >> > On Thu, Jun 7, 2018 at 4:09 PM, Nicolas Vazquez < >> > nicolas.vazq...@shapeblue.com> wrote: >> > >> > > Its format may depend on the provider, but under the hood it is a >> > snapshot >> > > which can be full or incremental and it is copied and stored in a >> backup >> > > repository in one of the backup servers. For example, if you use >> Veeam, >> > you >> > > will find your backups stored as .vbk (for full backups) and .vib (for >> > > incremental backups). It is the task of the backup provider to create >> and >> > > move these files to the backup repository. >> > > >> > > >> > > From: Rafael Weingärtner >> > > Sent: Thursday, June 7, 2018 10:39:35 AM >> > > To: dev >> > > Subject: Re: [PROPOSE] Backup and Recovery Framework >> > > >> > > Ok, there are a lot of things, which are kind of blurry to me. So, >> let’s >> > > try to go step by step. >> > > >> > > Let me ask again. What is a backup of the VM? You mention >> restore-point. >> > > What is it? Is it a snapshot in the hypervisor (complete/partial >> > snapshot)? >> > > If it is not a snapshot, what is it then? Are you installing some >> agent >> > in >> > > the VM and then copying/mirroring the files in the VM’s file system? >> > > >> > > On Thu, Jun 7, 2018 at 3:26 PM, Nicolas Vazquez < >> > > nicolas.vazq...@shapeblue.com> wrote: >> >
Re: [PROPOSE] Backup and Recovery Framework
Yes, this is important to note as most people don't understand this detail. It is VERY important to this discussion though. *Will Stevens* Chief Technology Officer c 514.826.0190 <https://goo.gl/NYZ8KK> On Thu, Jun 7, 2018 at 11:10 AM Todd Pigram wrote: > I agree completely. That is why I use inside VM backup products currently. > > On Thu, Jun 7, 2018 at 11:07 AM, Will Stevens > wrote: > > > I have not been following this thread entirely, but I would not count on > > our snapshot mechanism for backups. If something like Veeam is used, > then > > an agent is in play and that agent will quiesce the running processes, > such > > as MySQL. This is important in order to do backup and DR as you will > lose > > transactions in memory and can potentially have a corrupted backup. > > > > Snapshots are not backups!!! > > > > *Will Stevens* > > Chief Technology Officer > > c 514.826.0190 > > > > <https://goo.gl/NYZ8KK> > > > > > > On Thu, Jun 7, 2018 at 10:13 AM Rafael Weingärtner < > > rafaelweingart...@gmail.com> wrote: > > > > > Ok, now I start to understand. So, we will be handling snapshots and > > they > > > will be sent to a third-party system. > > > > > > Now let’s proceed. If I configure this “feature” with external=false. > > What > > > does it do? Nothing? I mean, what is the purpose of this “external” > > > parameter? > > > > > > > > > On Thu, Jun 7, 2018 at 4:09 PM, Nicolas Vazquez < > > > nicolas.vazq...@shapeblue.com> wrote: > > > > > > > Its format may depend on the provider, but under the hood it is a > > > snapshot > > > > which can be full or incremental and it is copied and stored in a > > backup > > > > repository in one of the backup servers. For example, if you use > Veeam, > > > you > > > > will find your backups stored as .vbk (for full backups) and .vib > (for > > > > incremental backups). It is the task of the backup provider to create > > and > > > > move these files to the backup repository. > > > > > > > > > > > > From: Rafael Weingärtner > > > > Sent: Thursday, June 7, 2018 10:39:35 AM > > > > To: dev > > > > Subject: Re: [PROPOSE] Backup and Recovery Framework > > > > > > > > Ok, there are a lot of things, which are kind of blurry to me. So, > > let’s > > > > try to go step by step. > > > > > > > > Let me ask again. What is a backup of the VM? You mention > > restore-point. > > > > What is it? Is it a snapshot in the hypervisor (complete/partial > > > snapshot)? > > > > If it is not a snapshot, what is it then? Are you installing some > > agent > > > in > > > > the VM and then copying/mirroring the files in the VM’s file system? > > > > > > > > On Thu, Jun 7, 2018 at 3:26 PM, Nicolas Vazquez < > > > > nicolas.vazq...@shapeblue.com> wrote: > > > > > > > > > Hi Daniel and Rafael, > > > > > > > > > > > > > > > I have updated the diagram picture on the FS, please let me know if > > the > > > > > problem persists. > > > > > > > > > > > > > > > @<mailto:daniel.herrm...@zv.fraunhofer.de>Daniel you are right > about > > > > > restoring VMs, please let me update the FS. Regarding your > question, > > we > > > > > have considered the backups as a first class citizens, meaning > that a > > > > > backup would continue existing unless its originating VM has been > > > > > removed/expunged. This way, listBackups API method would list them > in > > > > order > > > > > to allow a user to restore a deleted VM from a backup. > > > > > > > > > > > > > > > @Rafael the idea of the Dummy provider is simply returning the > > expected > > > > > entities from a backup provider. For example, the BackupManager > could > > > ask > > > > > the BackupProvider to list its backups and would expect a list of > > > > backups. > > > > > As the Dummy Provider implements the BackupProvider interface, it > has > > > its > > > > > own implementation of listBackups() method and simply returns some > > mock > > > > > entities to the Backup Manager. It d
Re: [PROPOSE] Backup and Recovery Framework
Now outside of our ACS, I use Veeam for vCloud and vSphere and use Alike for our XS pools. Inside ACS, the VMs use MaxBackup On Thu, Jun 7, 2018 at 11:09 AM, Todd Pigram wrote: > I agree completely. That is why I use inside VM backup products currently. > > On Thu, Jun 7, 2018 at 11:07 AM, Will Stevens > wrote: > >> I have not been following this thread entirely, but I would not count on >> our snapshot mechanism for backups. If something like Veeam is used, then >> an agent is in play and that agent will quiesce the running processes, >> such >> as MySQL. This is important in order to do backup and DR as you will lose >> transactions in memory and can potentially have a corrupted backup. >> >> Snapshots are not backups!!! >> >> *Will Stevens* >> Chief Technology Officer >> c 514.826.0190 >> >> <https://goo.gl/NYZ8KK> >> >> >> On Thu, Jun 7, 2018 at 10:13 AM Rafael Weingärtner < >> rafaelweingart...@gmail.com> wrote: >> >> > Ok, now I start to understand. So, we will be handling snapshots and >> they >> > will be sent to a third-party system. >> > >> > Now let’s proceed. If I configure this “feature” with external=false. >> What >> > does it do? Nothing? I mean, what is the purpose of this “external” >> > parameter? >> > >> > >> > On Thu, Jun 7, 2018 at 4:09 PM, Nicolas Vazquez < >> > nicolas.vazq...@shapeblue.com> wrote: >> > >> > > Its format may depend on the provider, but under the hood it is a >> > snapshot >> > > which can be full or incremental and it is copied and stored in a >> backup >> > > repository in one of the backup servers. For example, if you use >> Veeam, >> > you >> > > will find your backups stored as .vbk (for full backups) and .vib (for >> > > incremental backups). It is the task of the backup provider to create >> and >> > > move these files to the backup repository. >> > > >> > > >> > > From: Rafael Weingärtner >> > > Sent: Thursday, June 7, 2018 10:39:35 AM >> > > To: dev >> > > Subject: Re: [PROPOSE] Backup and Recovery Framework >> > > >> > > Ok, there are a lot of things, which are kind of blurry to me. So, >> let’s >> > > try to go step by step. >> > > >> > > Let me ask again. What is a backup of the VM? You mention >> restore-point. >> > > What is it? Is it a snapshot in the hypervisor (complete/partial >> > snapshot)? >> > > If it is not a snapshot, what is it then? Are you installing some >> agent >> > in >> > > the VM and then copying/mirroring the files in the VM’s file system? >> > > >> > > On Thu, Jun 7, 2018 at 3:26 PM, Nicolas Vazquez < >> > > nicolas.vazq...@shapeblue.com> wrote: >> > > >> > > > Hi Daniel and Rafael, >> > > > >> > > > >> > > > I have updated the diagram picture on the FS, please let me know if >> the >> > > > problem persists. >> > > > >> > > > >> > > > @<mailto:daniel.herrm...@zv.fraunhofer.de>Daniel you are right >> about >> > > > restoring VMs, please let me update the FS. Regarding your >> question, we >> > > > have considered the backups as a first class citizens, meaning that >> a >> > > > backup would continue existing unless its originating VM has been >> > > > removed/expunged. This way, listBackups API method would list them >> in >> > > order >> > > > to allow a user to restore a deleted VM from a backup. >> > > > >> > > > >> > > > @Rafael the idea of the Dummy provider is simply returning the >> expected >> > > > entities from a backup provider. For example, the BackupManager >> could >> > ask >> > > > the BackupProvider to list its backups and would expect a list of >> > > backups. >> > > > As the Dummy Provider implements the BackupProvider interface, it >> has >> > its >> > > > own implementation of listBackups() method and simply returns some >> mock >> > > > entities to the Backup Manager. It does not store any backup, it >> simply >> > > > creates them in execution time and returns them. >> > > > >> > > > >> &
Re: [PROPOSE] Backup and Recovery Framework
I agree completely. That is why I use inside VM backup products currently. On Thu, Jun 7, 2018 at 11:07 AM, Will Stevens wrote: > I have not been following this thread entirely, but I would not count on > our snapshot mechanism for backups. If something like Veeam is used, then > an agent is in play and that agent will quiesce the running processes, such > as MySQL. This is important in order to do backup and DR as you will lose > transactions in memory and can potentially have a corrupted backup. > > Snapshots are not backups!!! > > *Will Stevens* > Chief Technology Officer > c 514.826.0190 > > <https://goo.gl/NYZ8KK> > > > On Thu, Jun 7, 2018 at 10:13 AM Rafael Weingärtner < > rafaelweingart...@gmail.com> wrote: > > > Ok, now I start to understand. So, we will be handling snapshots and > they > > will be sent to a third-party system. > > > > Now let’s proceed. If I configure this “feature” with external=false. > What > > does it do? Nothing? I mean, what is the purpose of this “external” > > parameter? > > > > > > On Thu, Jun 7, 2018 at 4:09 PM, Nicolas Vazquez < > > nicolas.vazq...@shapeblue.com> wrote: > > > > > Its format may depend on the provider, but under the hood it is a > > snapshot > > > which can be full or incremental and it is copied and stored in a > backup > > > repository in one of the backup servers. For example, if you use Veeam, > > you > > > will find your backups stored as .vbk (for full backups) and .vib (for > > > incremental backups). It is the task of the backup provider to create > and > > > move these files to the backup repository. > > > > > > > > > From: Rafael Weingärtner > > > Sent: Thursday, June 7, 2018 10:39:35 AM > > > To: dev > > > Subject: Re: [PROPOSE] Backup and Recovery Framework > > > > > > Ok, there are a lot of things, which are kind of blurry to me. So, > let’s > > > try to go step by step. > > > > > > Let me ask again. What is a backup of the VM? You mention > restore-point. > > > What is it? Is it a snapshot in the hypervisor (complete/partial > > snapshot)? > > > If it is not a snapshot, what is it then? Are you installing some > agent > > in > > > the VM and then copying/mirroring the files in the VM’s file system? > > > > > > On Thu, Jun 7, 2018 at 3:26 PM, Nicolas Vazquez < > > > nicolas.vazq...@shapeblue.com> wrote: > > > > > > > Hi Daniel and Rafael, > > > > > > > > > > > > I have updated the diagram picture on the FS, please let me know if > the > > > > problem persists. > > > > > > > > > > > > @<mailto:daniel.herrm...@zv.fraunhofer.de>Daniel you are right about > > > > restoring VMs, please let me update the FS. Regarding your question, > we > > > > have considered the backups as a first class citizens, meaning that a > > > > backup would continue existing unless its originating VM has been > > > > removed/expunged. This way, listBackups API method would list them in > > > order > > > > to allow a user to restore a deleted VM from a backup. > > > > > > > > > > > > @Rafael the idea of the Dummy provider is simply returning the > expected > > > > entities from a backup provider. For example, the BackupManager could > > ask > > > > the BackupProvider to list its backups and would expect a list of > > > backups. > > > > As the Dummy Provider implements the BackupProvider interface, it has > > its > > > > own implementation of listBackups() method and simply returns some > mock > > > > entities to the Backup Manager. It does not store any backup, it > simply > > > > creates them in execution time and returns them. > > > > > > > > > > > > From the CloudStack point of view, Backups are stored externally, on > > the > > > > Backup and Recovery provider servers and are handled by them. We keep > > > track > > > > of the existing backups in the Backup provider. Implementation may > > depend > > > > on the Backup and Recovery provider, would consist on snapshots, but > > are > > > > handled by the provider and not by CloudStack. > > > > > > > > > > > > Backup policies are created on the Backup and Recovery provider and > > > > provide retention options and p
Re: [PROPOSE] Backup and Recovery Framework
I have not been following this thread entirely, but I would not count on our snapshot mechanism for backups. If something like Veeam is used, then an agent is in play and that agent will quiesce the running processes, such as MySQL. This is important in order to do backup and DR as you will lose transactions in memory and can potentially have a corrupted backup. Snapshots are not backups!!! *Will Stevens* Chief Technology Officer c 514.826.0190 <https://goo.gl/NYZ8KK> On Thu, Jun 7, 2018 at 10:13 AM Rafael Weingärtner < rafaelweingart...@gmail.com> wrote: > Ok, now I start to understand. So, we will be handling snapshots and they > will be sent to a third-party system. > > Now let’s proceed. If I configure this “feature” with external=false. What > does it do? Nothing? I mean, what is the purpose of this “external” > parameter? > > > On Thu, Jun 7, 2018 at 4:09 PM, Nicolas Vazquez < > nicolas.vazq...@shapeblue.com> wrote: > > > Its format may depend on the provider, but under the hood it is a > snapshot > > which can be full or incremental and it is copied and stored in a backup > > repository in one of the backup servers. For example, if you use Veeam, > you > > will find your backups stored as .vbk (for full backups) and .vib (for > > incremental backups). It is the task of the backup provider to create and > > move these files to the backup repository. > > > > > > From: Rafael Weingärtner > > Sent: Thursday, June 7, 2018 10:39:35 AM > > To: dev > > Subject: Re: [PROPOSE] Backup and Recovery Framework > > > > Ok, there are a lot of things, which are kind of blurry to me. So, let’s > > try to go step by step. > > > > Let me ask again. What is a backup of the VM? You mention restore-point. > > What is it? Is it a snapshot in the hypervisor (complete/partial > snapshot)? > > If it is not a snapshot, what is it then? Are you installing some agent > in > > the VM and then copying/mirroring the files in the VM’s file system? > > > > On Thu, Jun 7, 2018 at 3:26 PM, Nicolas Vazquez < > > nicolas.vazq...@shapeblue.com> wrote: > > > > > Hi Daniel and Rafael, > > > > > > > > > I have updated the diagram picture on the FS, please let me know if the > > > problem persists. > > > > > > > > > @<mailto:daniel.herrm...@zv.fraunhofer.de>Daniel you are right about > > > restoring VMs, please let me update the FS. Regarding your question, we > > > have considered the backups as a first class citizens, meaning that a > > > backup would continue existing unless its originating VM has been > > > removed/expunged. This way, listBackups API method would list them in > > order > > > to allow a user to restore a deleted VM from a backup. > > > > > > > > > @Rafael the idea of the Dummy provider is simply returning the expected > > > entities from a backup provider. For example, the BackupManager could > ask > > > the BackupProvider to list its backups and would expect a list of > > backups. > > > As the Dummy Provider implements the BackupProvider interface, it has > its > > > own implementation of listBackups() method and simply returns some mock > > > entities to the Backup Manager. It does not store any backup, it simply > > > creates them in execution time and returns them. > > > > > > > > > From the CloudStack point of view, Backups are stored externally, on > the > > > Backup and Recovery provider servers and are handled by them. We keep > > track > > > of the existing backups in the Backup provider. Implementation may > depend > > > on the Backup and Recovery provider, would consist on snapshots, but > are > > > handled by the provider and not by CloudStack. > > > > > > > > > Backup policies are created on the Backup and Recovery provider and > > > provide retention options and periods as well as backup times. For > > example, > > > the provider may expose Golden, Silver and Bronze policies and have > > > different options for each of them. > > > > > > > > > The 'external' parameter on the listBackupPolicies API method would > only > > > ask the Backup and Recovery provider to list their policies if it is > set > > to > > > true. If it is not set, or set to false, it just simply returns the > > > existing mapped policies in CloudStack. Initially, CloudStack does not > > > store any backup policy, those should be imported/map
Re: [PROPOSE] Backup and Recovery Framework
Rafael, I would agree with that as well as how Quadric spins up a ABD at time of backup. I have tried both Alike and Unitrends (old PHD virtual) and ACS removed the device before it works, but that was back on v4.3. On Thu, Jun 7, 2018 at 11:03 AM, Rafael Weingärtner < rafaelweingart...@gmail.com> wrote: > I believe the framework is going to be agnostic, but we need to implement > "driver" to deal with different vendors API, right? > > On Thu, Jun 7, 2018 at 5:01 PM, Todd Pigram wrote: > > > Nicolas, > > > > Given you Veeam example, this framework is backup vendor independent? So > I > > could use Veeam for vSphere or Hyper-V or use Quadric (Alike) for > XenServer > > or Hyper-V. This would sure beat the way I am doing it now, inside the > VM. > > > > I will be following this closely. > > > > > > > > On Thu, Jun 7, 2018 at 10:13 AM, Rafael Weingärtner < > > rafaelweingart...@gmail.com> wrote: > > > > > Ok, now I start to understand. So, we will be handling snapshots and > > they > > > will be sent to a third-party system. > > > > > > Now let’s proceed. If I configure this “feature” with external=false. > > What > > > does it do? Nothing? I mean, what is the purpose of this “external” > > > parameter? > > > > > > > > > On Thu, Jun 7, 2018 at 4:09 PM, Nicolas Vazquez < > > > nicolas.vazq...@shapeblue.com> wrote: > > > > > > > Its format may depend on the provider, but under the hood it is a > > > snapshot > > > > which can be full or incremental and it is copied and stored in a > > backup > > > > repository in one of the backup servers. For example, if you use > Veeam, > > > you > > > > will find your backups stored as .vbk (for full backups) and .vib > (for > > > > incremental backups). It is the task of the backup provider to create > > and > > > > move these files to the backup repository. > > > > > > > > > > > > From: Rafael Weingärtner > > > > Sent: Thursday, June 7, 2018 10:39:35 AM > > > > To: dev > > > > Subject: Re: [PROPOSE] Backup and Recovery Framework > > > > > > > > Ok, there are a lot of things, which are kind of blurry to me. So, > > let’s > > > > try to go step by step. > > > > > > > > Let me ask again. What is a backup of the VM? You mention > > restore-point. > > > > What is it? Is it a snapshot in the hypervisor (complete/partial > > > snapshot)? > > > > If it is not a snapshot, what is it then? Are you installing some > > agent > > > in > > > > the VM and then copying/mirroring the files in the VM’s file system? > > > > > > > > On Thu, Jun 7, 2018 at 3:26 PM, Nicolas Vazquez < > > > > nicolas.vazq...@shapeblue.com> wrote: > > > > > > > > > Hi Daniel and Rafael, > > > > > > > > > > > > > > > I have updated the diagram picture on the FS, please let me know if > > the > > > > > problem persists. > > > > > > > > > > > > > > > @<mailto:daniel.herrm...@zv.fraunhofer.de>Daniel you are right > about > > > > > restoring VMs, please let me update the FS. Regarding your > question, > > we > > > > > have considered the backups as a first class citizens, meaning > that a > > > > > backup would continue existing unless its originating VM has been > > > > > removed/expunged. This way, listBackups API method would list them > in > > > > order > > > > > to allow a user to restore a deleted VM from a backup. > > > > > > > > > > > > > > > @Rafael the idea of the Dummy provider is simply returning the > > expected > > > > > entities from a backup provider. For example, the BackupManager > could > > > ask > > > > > the BackupProvider to list its backups and would expect a list of > > > > backups. > > > > > As the Dummy Provider implements the BackupProvider interface, it > has > > > its > > > > > own implementation of listBackups() method and simply returns some > > mock > > > > > entities to the Backup Manager. It does not store any backup, it > > simply > > > > > creates them in execution time and returns them. > > > > > > >
Re: [PROPOSE] Backup and Recovery Framework
I believe the framework is going to be agnostic, but we need to implement "driver" to deal with different vendors API, right? On Thu, Jun 7, 2018 at 5:01 PM, Todd Pigram wrote: > Nicolas, > > Given you Veeam example, this framework is backup vendor independent? So I > could use Veeam for vSphere or Hyper-V or use Quadric (Alike) for XenServer > or Hyper-V. This would sure beat the way I am doing it now, inside the VM. > > I will be following this closely. > > > > On Thu, Jun 7, 2018 at 10:13 AM, Rafael Weingärtner < > rafaelweingart...@gmail.com> wrote: > > > Ok, now I start to understand. So, we will be handling snapshots and > they > > will be sent to a third-party system. > > > > Now let’s proceed. If I configure this “feature” with external=false. > What > > does it do? Nothing? I mean, what is the purpose of this “external” > > parameter? > > > > > > On Thu, Jun 7, 2018 at 4:09 PM, Nicolas Vazquez < > > nicolas.vazq...@shapeblue.com> wrote: > > > > > Its format may depend on the provider, but under the hood it is a > > snapshot > > > which can be full or incremental and it is copied and stored in a > backup > > > repository in one of the backup servers. For example, if you use Veeam, > > you > > > will find your backups stored as .vbk (for full backups) and .vib (for > > > incremental backups). It is the task of the backup provider to create > and > > > move these files to the backup repository. > > > > > > > > > From: Rafael Weingärtner > > > Sent: Thursday, June 7, 2018 10:39:35 AM > > > To: dev > > > Subject: Re: [PROPOSE] Backup and Recovery Framework > > > > > > Ok, there are a lot of things, which are kind of blurry to me. So, > let’s > > > try to go step by step. > > > > > > Let me ask again. What is a backup of the VM? You mention > restore-point. > > > What is it? Is it a snapshot in the hypervisor (complete/partial > > snapshot)? > > > If it is not a snapshot, what is it then? Are you installing some > agent > > in > > > the VM and then copying/mirroring the files in the VM’s file system? > > > > > > On Thu, Jun 7, 2018 at 3:26 PM, Nicolas Vazquez < > > > nicolas.vazq...@shapeblue.com> wrote: > > > > > > > Hi Daniel and Rafael, > > > > > > > > > > > > I have updated the diagram picture on the FS, please let me know if > the > > > > problem persists. > > > > > > > > > > > > @<mailto:daniel.herrm...@zv.fraunhofer.de>Daniel you are right about > > > > restoring VMs, please let me update the FS. Regarding your question, > we > > > > have considered the backups as a first class citizens, meaning that a > > > > backup would continue existing unless its originating VM has been > > > > removed/expunged. This way, listBackups API method would list them in > > > order > > > > to allow a user to restore a deleted VM from a backup. > > > > > > > > > > > > @Rafael the idea of the Dummy provider is simply returning the > expected > > > > entities from a backup provider. For example, the BackupManager could > > ask > > > > the BackupProvider to list its backups and would expect a list of > > > backups. > > > > As the Dummy Provider implements the BackupProvider interface, it has > > its > > > > own implementation of listBackups() method and simply returns some > mock > > > > entities to the Backup Manager. It does not store any backup, it > simply > > > > creates them in execution time and returns them. > > > > > > > > > > > > From the CloudStack point of view, Backups are stored externally, on > > the > > > > Backup and Recovery provider servers and are handled by them. We keep > > > track > > > > of the existing backups in the Backup provider. Implementation may > > depend > > > > on the Backup and Recovery provider, would consist on snapshots, but > > are > > > > handled by the provider and not by CloudStack. > > > > > > > > > > > > Backup policies are created on the Backup and Recovery provider and > > > > provide retention options and periods as well as backup times. For > > > example, > > > > the provider may expose Golden, Silver and Bronze policies and have > > > > dif
Re: [PROPOSE] Backup and Recovery Framework
Nicolas, Given you Veeam example, this framework is backup vendor independent? So I could use Veeam for vSphere or Hyper-V or use Quadric (Alike) for XenServer or Hyper-V. This would sure beat the way I am doing it now, inside the VM. I will be following this closely. On Thu, Jun 7, 2018 at 10:13 AM, Rafael Weingärtner < rafaelweingart...@gmail.com> wrote: > Ok, now I start to understand. So, we will be handling snapshots and they > will be sent to a third-party system. > > Now let’s proceed. If I configure this “feature” with external=false. What > does it do? Nothing? I mean, what is the purpose of this “external” > parameter? > > > On Thu, Jun 7, 2018 at 4:09 PM, Nicolas Vazquez < > nicolas.vazq...@shapeblue.com> wrote: > > > Its format may depend on the provider, but under the hood it is a > snapshot > > which can be full or incremental and it is copied and stored in a backup > > repository in one of the backup servers. For example, if you use Veeam, > you > > will find your backups stored as .vbk (for full backups) and .vib (for > > incremental backups). It is the task of the backup provider to create and > > move these files to the backup repository. > > > > > > From: Rafael Weingärtner > > Sent: Thursday, June 7, 2018 10:39:35 AM > > To: dev > > Subject: Re: [PROPOSE] Backup and Recovery Framework > > > > Ok, there are a lot of things, which are kind of blurry to me. So, let’s > > try to go step by step. > > > > Let me ask again. What is a backup of the VM? You mention restore-point. > > What is it? Is it a snapshot in the hypervisor (complete/partial > snapshot)? > > If it is not a snapshot, what is it then? Are you installing some agent > in > > the VM and then copying/mirroring the files in the VM’s file system? > > > > On Thu, Jun 7, 2018 at 3:26 PM, Nicolas Vazquez < > > nicolas.vazq...@shapeblue.com> wrote: > > > > > Hi Daniel and Rafael, > > > > > > > > > I have updated the diagram picture on the FS, please let me know if the > > > problem persists. > > > > > > > > > @<mailto:daniel.herrm...@zv.fraunhofer.de>Daniel you are right about > > > restoring VMs, please let me update the FS. Regarding your question, we > > > have considered the backups as a first class citizens, meaning that a > > > backup would continue existing unless its originating VM has been > > > removed/expunged. This way, listBackups API method would list them in > > order > > > to allow a user to restore a deleted VM from a backup. > > > > > > > > > @Rafael the idea of the Dummy provider is simply returning the expected > > > entities from a backup provider. For example, the BackupManager could > ask > > > the BackupProvider to list its backups and would expect a list of > > backups. > > > As the Dummy Provider implements the BackupProvider interface, it has > its > > > own implementation of listBackups() method and simply returns some mock > > > entities to the Backup Manager. It does not store any backup, it simply > > > creates them in execution time and returns them. > > > > > > > > > From the CloudStack point of view, Backups are stored externally, on > the > > > Backup and Recovery provider servers and are handled by them. We keep > > track > > > of the existing backups in the Backup provider. Implementation may > depend > > > on the Backup and Recovery provider, would consist on snapshots, but > are > > > handled by the provider and not by CloudStack. > > > > > > > > > Backup policies are created on the Backup and Recovery provider and > > > provide retention options and periods as well as backup times. For > > example, > > > the provider may expose Golden, Silver and Bronze policies and have > > > different options for each of them. > > > > > > > > > The 'external' parameter on the listBackupPolicies API method would > only > > > ask the Backup and Recovery provider to list their policies if it is > set > > to > > > true. If it is not set, or set to false, it just simply returns the > > > existing mapped policies in CloudStack. Initially, CloudStack does not > > > store any backup policy, those should be imported/mapped by the > > > 'importBackupPolicy' API method. > > > > > > > > > Finally, we consider backups as restore points. As mentioned above, > > > backups are stored on the Backup and Re
Re: [PROPOSE] Backup and Recovery Framework
Ok, now I start to understand. So, we will be handling snapshots and they will be sent to a third-party system. Now let’s proceed. If I configure this “feature” with external=false. What does it do? Nothing? I mean, what is the purpose of this “external” parameter? On Thu, Jun 7, 2018 at 4:09 PM, Nicolas Vazquez < nicolas.vazq...@shapeblue.com> wrote: > Its format may depend on the provider, but under the hood it is a snapshot > which can be full or incremental and it is copied and stored in a backup > repository in one of the backup servers. For example, if you use Veeam, you > will find your backups stored as .vbk (for full backups) and .vib (for > incremental backups). It is the task of the backup provider to create and > move these files to the backup repository. > > > From: Rafael Weingärtner > Sent: Thursday, June 7, 2018 10:39:35 AM > To: dev > Subject: Re: [PROPOSE] Backup and Recovery Framework > > Ok, there are a lot of things, which are kind of blurry to me. So, let’s > try to go step by step. > > Let me ask again. What is a backup of the VM? You mention restore-point. > What is it? Is it a snapshot in the hypervisor (complete/partial snapshot)? > If it is not a snapshot, what is it then? Are you installing some agent in > the VM and then copying/mirroring the files in the VM’s file system? > > On Thu, Jun 7, 2018 at 3:26 PM, Nicolas Vazquez < > nicolas.vazq...@shapeblue.com> wrote: > > > Hi Daniel and Rafael, > > > > > > I have updated the diagram picture on the FS, please let me know if the > > problem persists. > > > > > > @<mailto:daniel.herrm...@zv.fraunhofer.de>Daniel you are right about > > restoring VMs, please let me update the FS. Regarding your question, we > > have considered the backups as a first class citizens, meaning that a > > backup would continue existing unless its originating VM has been > > removed/expunged. This way, listBackups API method would list them in > order > > to allow a user to restore a deleted VM from a backup. > > > > > > @Rafael the idea of the Dummy provider is simply returning the expected > > entities from a backup provider. For example, the BackupManager could ask > > the BackupProvider to list its backups and would expect a list of > backups. > > As the Dummy Provider implements the BackupProvider interface, it has its > > own implementation of listBackups() method and simply returns some mock > > entities to the Backup Manager. It does not store any backup, it simply > > creates them in execution time and returns them. > > > > > > From the CloudStack point of view, Backups are stored externally, on the > > Backup and Recovery provider servers and are handled by them. We keep > track > > of the existing backups in the Backup provider. Implementation may depend > > on the Backup and Recovery provider, would consist on snapshots, but are > > handled by the provider and not by CloudStack. > > > > > > Backup policies are created on the Backup and Recovery provider and > > provide retention options and periods as well as backup times. For > example, > > the provider may expose Golden, Silver and Bronze policies and have > > different options for each of them. > > > > > > The 'external' parameter on the listBackupPolicies API method would only > > ask the Backup and Recovery provider to list their policies if it is set > to > > true. If it is not set, or set to false, it just simply returns the > > existing mapped policies in CloudStack. Initially, CloudStack does not > > store any backup policy, those should be imported/mapped by the > > 'importBackupPolicy' API method. > > > > > > Finally, we consider backups as restore points. As mentioned above, > > backups are stored on the Backup and Recovery provider servers, so we > keep > > a mapping between the CloudStack entity and the backup, which can be used > > to restore VMs. > > > > > > Please let me know if anything. > > > > > > Regards, > > > > Nicolas Vazquez > > > > > > From: Rafael Weingärtner > > Sent: Thursday, June 7, 2018 6:55:57 AM > > To: dev > > Subject: Re: [PROPOSE] Backup and Recovery Framework > > > > Hi Nicolas, > > > > The idea is great. I have some questions though. > > > > Before getting to the questions, your pictures are broken, or at least > they > > seem broken to me. > > > > How would the “dummy” backup provider work? I mean, what does it do? > where > > does it st
Re: [PROPOSE] Backup and Recovery Framework
Its format may depend on the provider, but under the hood it is a snapshot which can be full or incremental and it is copied and stored in a backup repository in one of the backup servers. For example, if you use Veeam, you will find your backups stored as .vbk (for full backups) and .vib (for incremental backups). It is the task of the backup provider to create and move these files to the backup repository. From: Rafael Weingärtner Sent: Thursday, June 7, 2018 10:39:35 AM To: dev Subject: Re: [PROPOSE] Backup and Recovery Framework Ok, there are a lot of things, which are kind of blurry to me. So, let’s try to go step by step. Let me ask again. What is a backup of the VM? You mention restore-point. What is it? Is it a snapshot in the hypervisor (complete/partial snapshot)? If it is not a snapshot, what is it then? Are you installing some agent in the VM and then copying/mirroring the files in the VM’s file system? On Thu, Jun 7, 2018 at 3:26 PM, Nicolas Vazquez < nicolas.vazq...@shapeblue.com> wrote: > Hi Daniel and Rafael, > > > I have updated the diagram picture on the FS, please let me know if the > problem persists. > > > @<mailto:daniel.herrm...@zv.fraunhofer.de>Daniel you are right about > restoring VMs, please let me update the FS. Regarding your question, we > have considered the backups as a first class citizens, meaning that a > backup would continue existing unless its originating VM has been > removed/expunged. This way, listBackups API method would list them in order > to allow a user to restore a deleted VM from a backup. > > > @Rafael the idea of the Dummy provider is simply returning the expected > entities from a backup provider. For example, the BackupManager could ask > the BackupProvider to list its backups and would expect a list of backups. > As the Dummy Provider implements the BackupProvider interface, it has its > own implementation of listBackups() method and simply returns some mock > entities to the Backup Manager. It does not store any backup, it simply > creates them in execution time and returns them. > > > From the CloudStack point of view, Backups are stored externally, on the > Backup and Recovery provider servers and are handled by them. We keep track > of the existing backups in the Backup provider. Implementation may depend > on the Backup and Recovery provider, would consist on snapshots, but are > handled by the provider and not by CloudStack. > > > Backup policies are created on the Backup and Recovery provider and > provide retention options and periods as well as backup times. For example, > the provider may expose Golden, Silver and Bronze policies and have > different options for each of them. > > > The 'external' parameter on the listBackupPolicies API method would only > ask the Backup and Recovery provider to list their policies if it is set to > true. If it is not set, or set to false, it just simply returns the > existing mapped policies in CloudStack. Initially, CloudStack does not > store any backup policy, those should be imported/mapped by the > 'importBackupPolicy' API method. > > > Finally, we consider backups as restore points. As mentioned above, > backups are stored on the Backup and Recovery provider servers, so we keep > a mapping between the CloudStack entity and the backup, which can be used > to restore VMs. > > > Please let me know if anything. > > > Regards, > > Nicolas Vazquez > > > From: Rafael Weingärtner > Sent: Thursday, June 7, 2018 6:55:57 AM > To: dev > Subject: Re: [PROPOSE] Backup and Recovery Framework > > Hi Nicolas, > > The idea is great. I have some questions though. > > Before getting to the questions, your pictures are broken, or at least they > seem broken to me. > > How would the “dummy” backup provider work? I mean, what does it do? where > does it storage the "backup"? > What is the difference from this Backup to a Snapshot? Is it simply storing > a snapshot outside the cloud structure? > How would these “backup policies” look like? I mean, what can I configure? > Regarding that parameter “external” of your backup policy, how would it > work if we set it to false? > What are you defining as a backup (what do you mean by backup in the > context of this feature?)? I mean, is it simply a snapshot file that you > store somewhere else? > > > nicolas.vazq...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > , > @shapeblue > > > nicolas.vazq...@shapeblue.com www.shapeblue.com , @shapeblue > On Thu, Jun 7, 2018 at 6:05 AM, wrote: > > > Hi all, > > > > +1 for this feature request, thank you for bringing this t
Re: [PROPOSE] Backup and Recovery Framework
Ok, there are a lot of things, which are kind of blurry to me. So, let’s try to go step by step. Let me ask again. What is a backup of the VM? You mention restore-point. What is it? Is it a snapshot in the hypervisor (complete/partial snapshot)? If it is not a snapshot, what is it then? Are you installing some agent in the VM and then copying/mirroring the files in the VM’s file system? On Thu, Jun 7, 2018 at 3:26 PM, Nicolas Vazquez < nicolas.vazq...@shapeblue.com> wrote: > Hi Daniel and Rafael, > > > I have updated the diagram picture on the FS, please let me know if the > problem persists. > > > @<mailto:daniel.herrm...@zv.fraunhofer.de>Daniel you are right about > restoring VMs, please let me update the FS. Regarding your question, we > have considered the backups as a first class citizens, meaning that a > backup would continue existing unless its originating VM has been > removed/expunged. This way, listBackups API method would list them in order > to allow a user to restore a deleted VM from a backup. > > > @Rafael the idea of the Dummy provider is simply returning the expected > entities from a backup provider. For example, the BackupManager could ask > the BackupProvider to list its backups and would expect a list of backups. > As the Dummy Provider implements the BackupProvider interface, it has its > own implementation of listBackups() method and simply returns some mock > entities to the Backup Manager. It does not store any backup, it simply > creates them in execution time and returns them. > > > From the CloudStack point of view, Backups are stored externally, on the > Backup and Recovery provider servers and are handled by them. We keep track > of the existing backups in the Backup provider. Implementation may depend > on the Backup and Recovery provider, would consist on snapshots, but are > handled by the provider and not by CloudStack. > > > Backup policies are created on the Backup and Recovery provider and > provide retention options and periods as well as backup times. For example, > the provider may expose Golden, Silver and Bronze policies and have > different options for each of them. > > > The 'external' parameter on the listBackupPolicies API method would only > ask the Backup and Recovery provider to list their policies if it is set to > true. If it is not set, or set to false, it just simply returns the > existing mapped policies in CloudStack. Initially, CloudStack does not > store any backup policy, those should be imported/mapped by the > 'importBackupPolicy' API method. > > > Finally, we consider backups as restore points. As mentioned above, > backups are stored on the Backup and Recovery provider servers, so we keep > a mapping between the CloudStack entity and the backup, which can be used > to restore VMs. > > > Please let me know if anything. > > > Regards, > > Nicolas Vazquez > > > From: Rafael Weingärtner > Sent: Thursday, June 7, 2018 6:55:57 AM > To: dev > Subject: Re: [PROPOSE] Backup and Recovery Framework > > Hi Nicolas, > > The idea is great. I have some questions though. > > Before getting to the questions, your pictures are broken, or at least they > seem broken to me. > > How would the “dummy” backup provider work? I mean, what does it do? where > does it storage the "backup"? > What is the difference from this Backup to a Snapshot? Is it simply storing > a snapshot outside the cloud structure? > How would these “backup policies” look like? I mean, what can I configure? > Regarding that parameter “external” of your backup policy, how would it > work if we set it to false? > What are you defining as a backup (what do you mean by backup in the > context of this feature?)? I mean, is it simply a snapshot file that you > store somewhere else? > > > nicolas.vazq...@shapeblue.com > www.shapeblue.com > , > @shapeblue > > > > On Thu, Jun 7, 2018 at 6:05 AM, wrote: > > > Hi all, > > > > +1 for this feature request, thank you for bringing this to life! Volume > > snapshots are what our users currently use as backup solution, which has > > some inherent problems such as handling large volumes and the handling of > > multiple volumes (consistency). > > > > One thought: when restoring a VM, I think there are two cases to > consider: > > > > - The VM is still in CS and the user just wants to restore an old version > > of the VM > > - The VM has already been deleted and the user wants to restore the > > deleted VM > > > > As listBackups is described as " listBackups API method. List existing > > backups for a VM". Is the second
Re: [PROPOSE] Backup and Recovery Framework
Hi Daniel and Rafael, I have updated the diagram picture on the FS, please let me know if the problem persists. @<mailto:daniel.herrm...@zv.fraunhofer.de>Daniel you are right about restoring VMs, please let me update the FS. Regarding your question, we have considered the backups as a first class citizens, meaning that a backup would continue existing unless its originating VM has been removed/expunged. This way, listBackups API method would list them in order to allow a user to restore a deleted VM from a backup. @Rafael the idea of the Dummy provider is simply returning the expected entities from a backup provider. For example, the BackupManager could ask the BackupProvider to list its backups and would expect a list of backups. As the Dummy Provider implements the BackupProvider interface, it has its own implementation of listBackups() method and simply returns some mock entities to the Backup Manager. It does not store any backup, it simply creates them in execution time and returns them. From the CloudStack point of view, Backups are stored externally, on the Backup and Recovery provider servers and are handled by them. We keep track of the existing backups in the Backup provider. Implementation may depend on the Backup and Recovery provider, would consist on snapshots, but are handled by the provider and not by CloudStack. Backup policies are created on the Backup and Recovery provider and provide retention options and periods as well as backup times. For example, the provider may expose Golden, Silver and Bronze policies and have different options for each of them. The 'external' parameter on the listBackupPolicies API method would only ask the Backup and Recovery provider to list their policies if it is set to true. If it is not set, or set to false, it just simply returns the existing mapped policies in CloudStack. Initially, CloudStack does not store any backup policy, those should be imported/mapped by the 'importBackupPolicy' API method. Finally, we consider backups as restore points. As mentioned above, backups are stored on the Backup and Recovery provider servers, so we keep a mapping between the CloudStack entity and the backup, which can be used to restore VMs. Please let me know if anything. Regards, Nicolas Vazquez From: Rafael Weingärtner Sent: Thursday, June 7, 2018 6:55:57 AM To: dev Subject: Re: [PROPOSE] Backup and Recovery Framework Hi Nicolas, The idea is great. I have some questions though. Before getting to the questions, your pictures are broken, or at least they seem broken to me. How would the “dummy” backup provider work? I mean, what does it do? where does it storage the "backup"? What is the difference from this Backup to a Snapshot? Is it simply storing a snapshot outside the cloud structure? How would these “backup policies” look like? I mean, what can I configure? Regarding that parameter “external” of your backup policy, how would it work if we set it to false? What are you defining as a backup (what do you mean by backup in the context of this feature?)? I mean, is it simply a snapshot file that you store somewhere else? nicolas.vazq...@shapeblue.com www.shapeblue.com , @shapeblue On Thu, Jun 7, 2018 at 6:05 AM, wrote: > Hi all, > > +1 for this feature request, thank you for bringing this to life! Volume > snapshots are what our users currently use as backup solution, which has > some inherent problems such as handling large volumes and the handling of > multiple volumes (consistency). > > One thought: when restoring a VM, I think there are two cases to consider: > > - The VM is still in CS and the user just wants to restore an old version > of the VM > - The VM has already been deleted and the user wants to restore the > deleted VM > > As listBackups is described as " listBackups API method. List existing > backups for a VM". Is the second use case supported at all? > > I like the approach not to define the backup policies in CS but in > whatever tool is used in the end, as otherwise a lot of features might be > lost. We as services providers need to create them in the first place, but > I think this is the better solution. > > Regards > Daniel > > On 06.06.18, 14:43, "Nicolas Vazquez" > wrote: > > Hi all, > > > We would like to introduce a new framework into CloudStack that will > allow adding Backup and Recovery providers as plugins. This framework goal > is allowing users to backup their guest VMs for recovery purposes. > > > Please find the FS on this link: https://cwiki.apache.org/ > confluence/display/CLOUDSTACK/Backup+and+Recovery+Framework > > > Regards, > > Nicolas Vazquez > > nicolas.vazq...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > , > @shapeblue > > > > > > -- Rafael Weingärtner
Re: [PROPOSE] Backup and Recovery Framework
Hi Nicolas, The idea is great. I have some questions though. Before getting to the questions, your pictures are broken, or at least they seem broken to me. How would the “dummy” backup provider work? I mean, what does it do? where does it storage the "backup"? What is the difference from this Backup to a Snapshot? Is it simply storing a snapshot outside the cloud structure? How would these “backup policies” look like? I mean, what can I configure? Regarding that parameter “external” of your backup policy, how would it work if we set it to false? What are you defining as a backup (what do you mean by backup in the context of this feature?)? I mean, is it simply a snapshot file that you store somewhere else? On Thu, Jun 7, 2018 at 6:05 AM, wrote: > Hi all, > > +1 for this feature request, thank you for bringing this to life! Volume > snapshots are what our users currently use as backup solution, which has > some inherent problems such as handling large volumes and the handling of > multiple volumes (consistency). > > One thought: when restoring a VM, I think there are two cases to consider: > > - The VM is still in CS and the user just wants to restore an old version > of the VM > - The VM has already been deleted and the user wants to restore the > deleted VM > > As listBackups is described as " listBackups API method. List existing > backups for a VM". Is the second use case supported at all? > > I like the approach not to define the backup policies in CS but in > whatever tool is used in the end, as otherwise a lot of features might be > lost. We as services providers need to create them in the first place, but > I think this is the better solution. > > Regards > Daniel > > On 06.06.18, 14:43, "Nicolas Vazquez" > wrote: > > Hi all, > > > We would like to introduce a new framework into CloudStack that will > allow adding Backup and Recovery providers as plugins. This framework goal > is allowing users to backup their guest VMs for recovery purposes. > > > Please find the FS on this link: https://cwiki.apache.org/ > confluence/display/CLOUDSTACK/Backup+and+Recovery+Framework > > > Regards, > > Nicolas Vazquez > > nicolas.vazq...@shapeblue.com > www.shapeblue.com > , > @shapeblue > > > > > > -- Rafael Weingärtner
Re: [PROPOSE] Backup and Recovery Framework
Hi all, +1 for this feature request, thank you for bringing this to life! Volume snapshots are what our users currently use as backup solution, which has some inherent problems such as handling large volumes and the handling of multiple volumes (consistency). One thought: when restoring a VM, I think there are two cases to consider: - The VM is still in CS and the user just wants to restore an old version of the VM - The VM has already been deleted and the user wants to restore the deleted VM As listBackups is described as " listBackups API method. List existing backups for a VM". Is the second use case supported at all? I like the approach not to define the backup policies in CS but in whatever tool is used in the end, as otherwise a lot of features might be lost. We as services providers need to create them in the first place, but I think this is the better solution. Regards Daniel On 06.06.18, 14:43, "Nicolas Vazquez" wrote: Hi all, We would like to introduce a new framework into CloudStack that will allow adding Backup and Recovery providers as plugins. This framework goal is allowing users to backup their guest VMs for recovery purposes. Please find the FS on this link: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Backup+and+Recovery+Framework Regards, Nicolas Vazquez nicolas.vazq...@shapeblue.com www.shapeblue.com , @shapeblue
[PROPOSE] Backup and Recovery Framework
Hi all, We would like to introduce a new framework into CloudStack that will allow adding Backup and Recovery providers as plugins. This framework goal is allowing users to backup their guest VMs for recovery purposes. Please find the FS on this link: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Backup+and+Recovery+Framework Regards, Nicolas Vazquez nicolas.vazq...@shapeblue.com www.shapeblue.com , @shapeblue
