Re: [openstack-dev] [nova] nova backup not working in stable/icehouse?
I also believe (2) is the most workable option. Full disclosure ... my current job is at EMC, and we just shipped a backup product for the VMware vCloud (the one used for VMware vCloud Air - http://vcloud.vmware.com/http://vcloud.vmware.com/). First release of that project was wrapping up, and I was asked to look at backup for OpenStack. As I am familiar with both open source community, and with how to do backup at cloud scale ... the problem is an easy fit. Different backup vendors might approach the problem in differing ways, which calls for a pluggable backend. What I find is that OpenStack is missing the hooks we need to do backup efficiently. There are promising proposed bits, but ... we are not there yet. Storage for backups is of a different character than existing services in OpenStack. Backup vendors need a place to plug in, and need some relevant primitive operations. Turns out the AWS folk already have rather nice support for backup from a cloud developer's perspective. At present, there is nothing equivalent in OpenStack. From a cloud developer's perspective, that is a huge lack. While AWS is an API wrapped around a single service, OpenStack is an API wrapped around differing services. This is both harder (in terms of defining the API), and an advantage to customers with differing requirements. What lacks and is needed by backup vendors is the right set of primitives. On Sun, Aug 31, 2014 at 8:19 AM, laserjetyang wrote: > I tend to say 2) is the best option. There are many open source or > commercial backup software, and both for VMs and volume. > If we do option 1), it reminds me to implement something similar to VMware > method, and it will cause nova really heavy. > > > On Sun, Aug 31, 2014 at 4:04 AM, Preston L. Bannister < > pres...@bannister.us> wrote: > >> You are thinking of written-for-cloud applications. For those the state >> should not persist with the instance. >> >> There are a very large number of existing applications, not written to >> the cloud model, but which could be deployed in a cloud. Those applications >> are not all going to get re-written (as the cost is often greater than the >> benefit). Those applications need some ready and efficient means of backup. >> >> The benefits of the cloud-application model and the cloud-deployment >> model are distinct. >> >> The existing nova backup (if it worked) is an inefficient snapshot. Not >> really useful at scale. >> >> There are two basic paths forward, here. 1) Build a complete common >> backup implementation that everyone can use. Or 2) define a common API for >> invoking backup, allow vendors to supply differing implementations, and add >> to OpenStack the APIs needed by backup implementations. >> >> Given past history, there does not seem to be enough focus or resources >> to get (1) done. >> >> That makes (2) much more likely. Reasonably sure we can find the interest >> and resources for this path. :) >> >> >> >> >> >> >> On Fri, Aug 29, 2014 at 10:55 PM, laserjetyang >> wrote: >> >>> I think the purpose of nova VM is not for persistent usage, and it >>> should be used for stateless. However, there are use cases to use VM to >>> replace bare metal applications, and it requires the same coverage, which I >>> think VMware did pretty well. >>> The nova backup is snapshot indeed, so it should be re-implemented to be >>> fitting into the real backup solution. >>> >>> >>> On Sat, Aug 30, 2014 at 1:14 PM, Preston L. Bannister < >>> pres...@bannister.us> wrote: >>> The current "backup" APIs in OpenStack do not really make sense (and apparently do not work ... which perhaps says something about usage and usability). So in that sense, they could be removed. Wrote out a bit as to what is needed: http://bannister.us/weblog/2014/08/21/cloud-application-backup-and-openstack/ At the same time, to do efficient backup at cloud scale, OpenStack is missing a few primitives needed for backup. We need to be able to quiesce instances, and collect changed-block lists, across hypervisors and filesystems. There is some relevant work in this area - for example: https://wiki.openstack.org/wiki/Nova/InstanceLevelSnapshots Switching hats - as a cloud developer, on AWS there is excellent current means of backup-through-snapshots, which is very quick and is charged based on changed-blocks. (The performance and cost both reflect use of changed-block tracking underneath.) If OpenStack completely lacks any equivalent API, then the platform is less competitive. Are you thinking about backup as performed by the cloud infrastructure folk, or as a service used by cloud developers in deployed applications? The first might do behind-the-scenes backups. The second needs an API. On Fri, Aug 29, 2014 at 11:16 AM, Jay Pipes wrote: > On 08/29/2014 02:48 AM, Preston L. Bannister wrote:
Re: [openstack-dev] [nova] nova backup not working in stable/icehouse?
I tend to say 2) is the best option. There are many open source or commercial backup software, and both for VMs and volume. If we do option 1), it reminds me to implement something similar to VMware method, and it will cause nova really heavy. On Sun, Aug 31, 2014 at 4:04 AM, Preston L. Bannister wrote: > You are thinking of written-for-cloud applications. For those the state > should not persist with the instance. > > There are a very large number of existing applications, not written to the > cloud model, but which could be deployed in a cloud. Those applications are > not all going to get re-written (as the cost is often greater than the > benefit). Those applications need some ready and efficient means of backup. > > The benefits of the cloud-application model and the cloud-deployment model > are distinct. > > The existing nova backup (if it worked) is an inefficient snapshot. Not > really useful at scale. > > There are two basic paths forward, here. 1) Build a complete common > backup implementation that everyone can use. Or 2) define a common API for > invoking backup, allow vendors to supply differing implementations, and add > to OpenStack the APIs needed by backup implementations. > > Given past history, there does not seem to be enough focus or resources to > get (1) done. > > That makes (2) much more likely. Reasonably sure we can find the interest > and resources for this path. :) > > > > > > > On Fri, Aug 29, 2014 at 10:55 PM, laserjetyang > wrote: > >> I think the purpose of nova VM is not for persistent usage, and it should >> be used for stateless. However, there are use cases to use VM to replace >> bare metal applications, and it requires the same coverage, which I think >> VMware did pretty well. >> The nova backup is snapshot indeed, so it should be re-implemented to be >> fitting into the real backup solution. >> >> >> On Sat, Aug 30, 2014 at 1:14 PM, Preston L. Bannister < >> pres...@bannister.us> wrote: >> >>> The current "backup" APIs in OpenStack do not really make sense (and >>> apparently do not work ... which perhaps says something about usage and >>> usability). So in that sense, they could be removed. >>> >>> Wrote out a bit as to what is needed: >>> >>> http://bannister.us/weblog/2014/08/21/cloud-application-backup-and-openstack/ >>> >>> At the same time, to do efficient backup at cloud scale, OpenStack is >>> missing a few primitives needed for backup. We need to be able to quiesce >>> instances, and collect changed-block lists, across hypervisors and >>> filesystems. There is some relevant work in this area - for example: >>> >>> https://wiki.openstack.org/wiki/Nova/InstanceLevelSnapshots >>> >>> Switching hats - as a cloud developer, on AWS there is excellent current >>> means of backup-through-snapshots, which is very quick and is charged based >>> on changed-blocks. (The performance and cost both reflect use of >>> changed-block tracking underneath.) >>> >>> If OpenStack completely lacks any equivalent API, then the platform is >>> less competitive. >>> >>> Are you thinking about backup as performed by the cloud infrastructure >>> folk, or as a service used by cloud developers in deployed applications? >>> The first might do behind-the-scenes backups. The second needs an API. >>> >>> >>> >>> >>> On Fri, Aug 29, 2014 at 11:16 AM, Jay Pipes wrote: >>> On 08/29/2014 02:48 AM, Preston L. Bannister wrote: > Looking to put a proper implementation of instance backup into > OpenStack. Started by writing a simple set of baseline tests and > running > against the stable/icehouse branch. They failed! > > https://github.com/dreadedhill-work/openstack-backup-scripts > > Scripts and configuration are in the above. Simple tests. > > At first I assumed there was a configuration error in my Devstack ... > but at this point I believe the errors are in fact in OpenStack. (Also > I > have rather more colorful things to say about what is and is not > logged.) > > Try to backup bootable Cinder volumes attached to instances ... and all > fail. Try to backup instances booted from images, and all-but-one fail > (without logged errors, so far as I see). > > Was concerned about preserving existing behaviour (as I am currently > hacking the Nova backup API), but ... if the existing is badly broken, > this may not be a concern. (Makes my job a bit simpler.) > > If someone is using "nova backup" successfully (more than one backup at > a time), I *would* rather like to know! > > Anyone with different experience? > IMO, the create_backup API extension should be removed from the Compute API. It's completely unnecessary and backups should be the purview of external (to Nova) scripts or configuration management modules. This API extension is essentially trying to be a Cloud Cron, which is inappropriate for the Compute API, IMO. -jay >
Re: [openstack-dev] [nova] nova backup not working in stable/icehouse?
You are thinking of written-for-cloud applications. For those the state should not persist with the instance. There are a very large number of existing applications, not written to the cloud model, but which could be deployed in a cloud. Those applications are not all going to get re-written (as the cost is often greater than the benefit). Those applications need some ready and efficient means of backup. The benefits of the cloud-application model and the cloud-deployment model are distinct. The existing nova backup (if it worked) is an inefficient snapshot. Not really useful at scale. There are two basic paths forward, here. 1) Build a complete common backup implementation that everyone can use. Or 2) define a common API for invoking backup, allow vendors to supply differing implementations, and add to OpenStack the APIs needed by backup implementations. Given past history, there does not seem to be enough focus or resources to get (1) done. That makes (2) much more likely. Reasonably sure we can find the interest and resources for this path. :) On Fri, Aug 29, 2014 at 10:55 PM, laserjetyang wrote: > I think the purpose of nova VM is not for persistent usage, and it should > be used for stateless. However, there are use cases to use VM to replace > bare metal applications, and it requires the same coverage, which I think > VMware did pretty well. > The nova backup is snapshot indeed, so it should be re-implemented to be > fitting into the real backup solution. > > > On Sat, Aug 30, 2014 at 1:14 PM, Preston L. Bannister < > pres...@bannister.us> wrote: > >> The current "backup" APIs in OpenStack do not really make sense (and >> apparently do not work ... which perhaps says something about usage and >> usability). So in that sense, they could be removed. >> >> Wrote out a bit as to what is needed: >> >> http://bannister.us/weblog/2014/08/21/cloud-application-backup-and-openstack/ >> >> At the same time, to do efficient backup at cloud scale, OpenStack is >> missing a few primitives needed for backup. We need to be able to quiesce >> instances, and collect changed-block lists, across hypervisors and >> filesystems. There is some relevant work in this area - for example: >> >> https://wiki.openstack.org/wiki/Nova/InstanceLevelSnapshots >> >> Switching hats - as a cloud developer, on AWS there is excellent current >> means of backup-through-snapshots, which is very quick and is charged based >> on changed-blocks. (The performance and cost both reflect use of >> changed-block tracking underneath.) >> >> If OpenStack completely lacks any equivalent API, then the platform is >> less competitive. >> >> Are you thinking about backup as performed by the cloud infrastructure >> folk, or as a service used by cloud developers in deployed applications? >> The first might do behind-the-scenes backups. The second needs an API. >> >> >> >> >> On Fri, Aug 29, 2014 at 11:16 AM, Jay Pipes wrote: >> >>> On 08/29/2014 02:48 AM, Preston L. Bannister wrote: >>> Looking to put a proper implementation of instance backup into OpenStack. Started by writing a simple set of baseline tests and running against the stable/icehouse branch. They failed! https://github.com/dreadedhill-work/openstack-backup-scripts Scripts and configuration are in the above. Simple tests. At first I assumed there was a configuration error in my Devstack ... but at this point I believe the errors are in fact in OpenStack. (Also I have rather more colorful things to say about what is and is not logged.) Try to backup bootable Cinder volumes attached to instances ... and all fail. Try to backup instances booted from images, and all-but-one fail (without logged errors, so far as I see). Was concerned about preserving existing behaviour (as I am currently hacking the Nova backup API), but ... if the existing is badly broken, this may not be a concern. (Makes my job a bit simpler.) If someone is using "nova backup" successfully (more than one backup at a time), I *would* rather like to know! Anyone with different experience? >>> >>> IMO, the create_backup API extension should be removed from the Compute >>> API. It's completely unnecessary and backups should be the purview of >>> external (to Nova) scripts or configuration management modules. This API >>> extension is essentially trying to be a Cloud Cron, which is inappropriate >>> for the Compute API, IMO. >>> >>> -jay >>> >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > ___ > Ope
Re: [openstack-dev] [nova] nova backup not working in stable/icehouse?
I think the purpose of nova VM is not for persistent usage, and it should be used for stateless. However, there are use cases to use VM to replace bare metal applications, and it requires the same coverage, which I think VMware did pretty well. The nova backup is snapshot indeed, so it should be re-implemented to be fitting into the real backup solution. On Sat, Aug 30, 2014 at 1:14 PM, Preston L. Bannister wrote: > The current "backup" APIs in OpenStack do not really make sense (and > apparently do not work ... which perhaps says something about usage and > usability). So in that sense, they could be removed. > > Wrote out a bit as to what is needed: > > http://bannister.us/weblog/2014/08/21/cloud-application-backup-and-openstack/ > > At the same time, to do efficient backup at cloud scale, OpenStack is > missing a few primitives needed for backup. We need to be able to quiesce > instances, and collect changed-block lists, across hypervisors and > filesystems. There is some relevant work in this area - for example: > > https://wiki.openstack.org/wiki/Nova/InstanceLevelSnapshots > > Switching hats - as a cloud developer, on AWS there is excellent current > means of backup-through-snapshots, which is very quick and is charged based > on changed-blocks. (The performance and cost both reflect use of > changed-block tracking underneath.) > > If OpenStack completely lacks any equivalent API, then the platform is > less competitive. > > Are you thinking about backup as performed by the cloud infrastructure > folk, or as a service used by cloud developers in deployed applications? > The first might do behind-the-scenes backups. The second needs an API. > > > > > On Fri, Aug 29, 2014 at 11:16 AM, Jay Pipes wrote: > >> On 08/29/2014 02:48 AM, Preston L. Bannister wrote: >> >>> Looking to put a proper implementation of instance backup into >>> OpenStack. Started by writing a simple set of baseline tests and running >>> against the stable/icehouse branch. They failed! >>> >>> https://github.com/dreadedhill-work/openstack-backup-scripts >>> >>> Scripts and configuration are in the above. Simple tests. >>> >>> At first I assumed there was a configuration error in my Devstack ... >>> but at this point I believe the errors are in fact in OpenStack. (Also I >>> have rather more colorful things to say about what is and is not logged.) >>> >>> Try to backup bootable Cinder volumes attached to instances ... and all >>> fail. Try to backup instances booted from images, and all-but-one fail >>> (without logged errors, so far as I see). >>> >>> Was concerned about preserving existing behaviour (as I am currently >>> hacking the Nova backup API), but ... if the existing is badly broken, >>> this may not be a concern. (Makes my job a bit simpler.) >>> >>> If someone is using "nova backup" successfully (more than one backup at >>> a time), I *would* rather like to know! >>> >>> Anyone with different experience? >>> >> >> IMO, the create_backup API extension should be removed from the Compute >> API. It's completely unnecessary and backups should be the purview of >> external (to Nova) scripts or configuration management modules. This API >> extension is essentially trying to be a Cloud Cron, which is inappropriate >> for the Compute API, IMO. >> >> -jay >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] nova backup not working in stable/icehouse?
The current "backup" APIs in OpenStack do not really make sense (and apparently do not work ... which perhaps says something about usage and usability). So in that sense, they could be removed. Wrote out a bit as to what is needed: http://bannister.us/weblog/2014/08/21/cloud-application-backup-and-openstack/ At the same time, to do efficient backup at cloud scale, OpenStack is missing a few primitives needed for backup. We need to be able to quiesce instances, and collect changed-block lists, across hypervisors and filesystems. There is some relevant work in this area - for example: https://wiki.openstack.org/wiki/Nova/InstanceLevelSnapshots Switching hats - as a cloud developer, on AWS there is excellent current means of backup-through-snapshots, which is very quick and is charged based on changed-blocks. (The performance and cost both reflect use of changed-block tracking underneath.) If OpenStack completely lacks any equivalent API, then the platform is less competitive. Are you thinking about backup as performed by the cloud infrastructure folk, or as a service used by cloud developers in deployed applications? The first might do behind-the-scenes backups. The second needs an API. On Fri, Aug 29, 2014 at 11:16 AM, Jay Pipes wrote: > On 08/29/2014 02:48 AM, Preston L. Bannister wrote: > >> Looking to put a proper implementation of instance backup into >> OpenStack. Started by writing a simple set of baseline tests and running >> against the stable/icehouse branch. They failed! >> >> https://github.com/dreadedhill-work/openstack-backup-scripts >> >> Scripts and configuration are in the above. Simple tests. >> >> At first I assumed there was a configuration error in my Devstack ... >> but at this point I believe the errors are in fact in OpenStack. (Also I >> have rather more colorful things to say about what is and is not logged.) >> >> Try to backup bootable Cinder volumes attached to instances ... and all >> fail. Try to backup instances booted from images, and all-but-one fail >> (without logged errors, so far as I see). >> >> Was concerned about preserving existing behaviour (as I am currently >> hacking the Nova backup API), but ... if the existing is badly broken, >> this may not be a concern. (Makes my job a bit simpler.) >> >> If someone is using "nova backup" successfully (more than one backup at >> a time), I *would* rather like to know! >> >> Anyone with different experience? >> > > IMO, the create_backup API extension should be removed from the Compute > API. It's completely unnecessary and backups should be the purview of > external (to Nova) scripts or configuration management modules. This API > extension is essentially trying to be a Cloud Cron, which is inappropriate > for the Compute API, IMO. > > -jay > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] nova backup not working in stable/icehouse?
On 08/29/2014 02:48 AM, Preston L. Bannister wrote: Looking to put a proper implementation of instance backup into OpenStack. Started by writing a simple set of baseline tests and running against the stable/icehouse branch. They failed! https://github.com/dreadedhill-work/openstack-backup-scripts Scripts and configuration are in the above. Simple tests. At first I assumed there was a configuration error in my Devstack ... but at this point I believe the errors are in fact in OpenStack. (Also I have rather more colorful things to say about what is and is not logged.) Try to backup bootable Cinder volumes attached to instances ... and all fail. Try to backup instances booted from images, and all-but-one fail (without logged errors, so far as I see). Was concerned about preserving existing behaviour (as I am currently hacking the Nova backup API), but ... if the existing is badly broken, this may not be a concern. (Makes my job a bit simpler.) If someone is using "nova backup" successfully (more than one backup at a time), I *would* rather like to know! Anyone with different experience? IMO, the create_backup API extension should be removed from the Compute API. It's completely unnecessary and backups should be the purview of external (to Nova) scripts or configuration management modules. This API extension is essentially trying to be a Cloud Cron, which is inappropriate for the Compute API, IMO. -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
