Re: [openstack-dev] [Freezer] PTL Candidacy
And you have at least all my support : ) On Wed, Jan 25, 2017 at 12:56 PM, Saad Zaher wrote: > Hello everyone, > > I'm happy to announce my candidacy to be the Freezer PTL for the Pike > release > cycle. > > The Freezer developers did a very good job over the past releases and I am > sure > they will continue doing an amazing job over the next release as well. > > For the Pike release cycle, I think we need to focus more on backing up > OpenStack resources and adding more > backup engines to give more variety to OpenStack users, also we should > continue working on > the core freezer features to provide stabilization. > > In this cycle we will give also more attention to freezer-dr, the disaster > recovery > part of freezer. It provides cloud administrators the capability to > evacuate > VMs from failed compute nodes, we aim to take it to more than that and > support more severe disaster. > > > In order to achieve that, I think we should focus on: > > * Enhancing core freezer-agent features and continue refactoring the > missing parts > to allow pluggable architecture to support more engines and applications > in freezer. > > * Move from Elasticsearch to Oslo.db > > * Fully implement Oslo.policy > > * Implement version 2 of freezer-api > > * Integration tests. We need to increase the work done on testing. This > will > help to stabilize Freezer. > > * Documentation. We should target for a split, refactoring and global > improvement of our docs, which is a required step to increase the size > of our > community. > > * Focus on implementing OpenStack engine(s) to backup OpenStack resources. > > > I would be honoured to have your support. > > Thanks, > Saad Zaher (szaher) > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Freezer] Replace Gnu Tar with DAR
If that would work, the only difference would probably be that when restoring, the deleted files between levels are restore too. No difference for incremental backups as --listed-incremental is still used. So when the restore of that level fails, it is restarted without --incremental related options. It is a possible workaround. A possible solution would be to not wrap binaries any more and provide similar functionalities from python code, so the code would also be more portable. I'd rather invest time on binary dependency removal rather than use other binaries. Just my opinion. On Mon, May 23, 2016 at 6:03 PM, Dieterly, Deklan wrote: > Then it would not be an incremental backup/restore. This problem arises > when doing incremental backup and restores. > -- > Deklan Dieterly > > Senior Systems Software Engineer > HPE > > > > > From: Fausto Marzi > Reply-To: "OpenStack Development Mailing List (not for usage questions)" > > Date: Wednesday, May 18, 2016 at 5:22 AM > To: "OpenStack Development Mailing List (not for usage questions)" > > Subject: Re: [openstack-dev] [Freezer] Replace Gnu Tar with DAR > > > >Hi Deklan, > > > >what happen if the extract is executed without --listed-incremental or > >--incremental options? > > > > > >Does the issue still happen? > > > > > >Thanks, > > > >Fausto > > > > > >On Sat, May 14, 2016 at 12:56 AM, Dieterly, Deklan > > wrote: > > > >When using incremental backups, tar will not handle removing a dir and > >then renaming another dir to the removed dir. > > > > > >dek@dek-HP-Z620-Workstation:~/backup-test$ tar --extract > >--listed-incrementa=/dev/null --file backup.2.tar > >tar: Cannot rename Œbackup/dir1¹ to Œbackup/dir2¹: Directory not empty > >tar: Exiting with failure status due to previous errors > > > > > > > >Here are the steps to reproduce. > > > > 1845 mkdir backup > > 1846 mkdir backup/dir1 > > 1847 mkdir backup/dir2 > > 1848 echo "aa" > backup/dir1/dir1-file1 > > 1849 echo "aa" > backup/dir2/dir2-file1 > > 1852 tar --create --file=backup.tar --listed-incremental=./listed-incr > >backup > > 1854 rm -rf backup/dir2 > > 1855 mv backup/dir1 backup/dir2 > > 1856 tar --create --file=backup.2.tar --listed-incremental=./listed-incr > >backup > > 1859 tar --extract --listed-incrementa=/dev/null --file backup.tar > > 1861 tar --extract --listed-incrementa=/dev/null --file backup.2.tar > > > > > >This seems to be a well known, long-standing issue with tar. > >-- > >Deklan Dieterly > > > >Senior Systems Software Engineer > >HPE > > > > > > > > > >On 5/13/16, 4:33 PM, "Fox, Kevin M" wrote: > > > >>Whats the issue? > >> > >>From: Dieterly, Deklan [deklan.diete...@hpe.com] > >>Sent: Friday, May 13, 2016 3:07 PM > >>To: openstack-dev@lists.openstack.org > >>Subject: [openstack-dev] [Freezer] Replace Gnu Tar with DAR > >> > >>Does anybody see any issues if Freezer used DAR instead of Gnu Tar? DAR > >>seems to handle a particular use case that Freezer has while Gnu Tar does > >>not. > >>-- > >>Deklan Dieterly > >> > >>Senior Systems Software Engineer > >>HPE > >> > >> > >>_ > >>_ > >>OpenStack Development Mailing List (not for usage questions) > >>Unsubscribe: > >openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > ><http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >>_ > >>_ > >>OpenStack Development Mailing List (not for usage questions) > >>Unsubscribe: > >openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > ><http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > >__ > >OpenStack Development Mailing List (not for usage questions) > >Unsubscribe: > >openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > ><http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Freezer] Replace Gnu Tar with DAR
Hi Deklan, what happen if the extract is executed without --listed-incremental or --incremental options? Does the issue still happen? Thanks, Fausto On Sat, May 14, 2016 at 12:56 AM, Dieterly, Deklan wrote: > When using incremental backups, tar will not handle removing a dir and > then renaming another dir to the removed dir. > > > dek@dek-HP-Z620-Workstation:~/backup-test$ tar --extract > --listed-incrementa=/dev/null --file backup.2.tar > tar: Cannot rename Œbackup/dir1¹ to Œbackup/dir2¹: Directory not empty > tar: Exiting with failure status due to previous errors > > > > Here are the steps to reproduce. > > 1845 mkdir backup > 1846 mkdir backup/dir1 > 1847 mkdir backup/dir2 > 1848 echo "aa" > backup/dir1/dir1-file1 > 1849 echo "aa" > backup/dir2/dir2-file1 > 1852 tar --create --file=backup.tar --listed-incremental=./listed-incr > backup > 1854 rm -rf backup/dir2 > 1855 mv backup/dir1 backup/dir2 > 1856 tar --create --file=backup.2.tar --listed-incremental=./listed-incr > backup > 1859 tar --extract --listed-incrementa=/dev/null --file backup.tar > 1861 tar --extract --listed-incrementa=/dev/null --file backup.2.tar > > > This seems to be a well known, long-standing issue with tar. > -- > Deklan Dieterly > > Senior Systems Software Engineer > HPE > > > > > On 5/13/16, 4:33 PM, "Fox, Kevin M" wrote: > > >Whats the issue? > > > >From: Dieterly, Deklan [deklan.diete...@hpe.com] > >Sent: Friday, May 13, 2016 3:07 PM > >To: openstack-dev@lists.openstack.org > >Subject: [openstack-dev] [Freezer] Replace Gnu Tar with DAR > > > >Does anybody see any issues if Freezer used DAR instead of Gnu Tar? DAR > >seems to handle a particular use case that Freezer has while Gnu Tar does > >not. > >-- > >Deklan Dieterly > > > >Senior Systems Software Engineer > >HPE > > > > > >__ > >OpenStack Development Mailing List (not for usage questions) > >Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > >__ > >OpenStack Development Mailing List (not for usage questions) > >Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] supporting Go
++ Brilliant. On Fri, May 13, 2016 at 10:14 AM, Dmitry Tantsur wrote: > > This is pretty subjective, I would say. I personally don't feel Go > (especially its approach to error handling) any natural (at least no more > than Rust or Scala, for example). If familiarity for Python developers is > an argument here, mastering Cython or making OpenStack run on PyPy must be > much easier for a random Python developer out there to seriously bump the > performance. And it would not require introducing a completely new language > to the picture. > > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] supporting Go
My take would be to select no more than 3 languages, according what they are needed for, then let the Service Team pick the best one right for what it needs to be done. Something like: - Do you need more performance for this component in your service? OK, use this. - Do you need Web and alike? Ok, use that. - General and Default: Python. - Anything else? Then you can use this. With this approach we would cover the needs of developers that have to implement services and at the same time provide options for languages that are better suited for related use cases. All the gating, tests and alike will be focused on those selected languages. This would also solve discussions in the future, as if for instance a new language is proposed for performances, then we would already have a solution for that. At the same time we would have an open approach to alternatives, but only to those alternatives, not any and still be able to keep control and focus. Thanks, Fausto On Thu, May 12, 2016 at 9:04 PM, Robert Collins wrote: > On 13 May 2016 at 00:01, Thierry Carrez wrote: > > Robert Collins wrote: > >> > >> [...] > >> So, given that that is the model - why is language part of it? Yes, > >> there are minimum overheads to having a given language in CI [...] > > > > > > By "minimum" do you mean that they are someone else's problem ? > > No. I mean that there is an irreducible cost: there is some minimum > and we need to account for that. (as in fact the end of that paragraph > which you snipped, said). > > > There are economics at play here. Adding a language simplifies the work > of > > some and make more work for others. Obviously "some" see mostly benefits > and > > "others" see mostly drawbacks. You're just creating an externality[1]. > > I'm not sure that allowing arbitrary languages would constitute an > externality, properly structured - which is what I was trying to get > at with my mail. > > > So rather than shifting workloads to someone else or pretending there is > no > > problem, let's take the time to calmly measure the cost, see what > resources > > we have lined up to address that cost, and make a decision from there. > > I proposed neither of those things (shifting, or pretence). Rebutting > those positions is not rebutting my argument. > > I agree with taking the time to assess things carefully, but in this > discussion noone had taken a general 'pro' stance, so I decided, as an > exercise, to write one up. > > However, assessing 'what we have to address that cost' is missing the > actual point: we don't need to cover the cost /once/, we need to make > how-its-covered, and who-covers-it be structured such that folk > wanting to use $LANGUAGE provide the [human] resources needed to cover > the incurred costs. An answer which says 'we can just afford to pay > for go, but thats it', is an answer that has failed to actually tackle > the underlying problem. > > -Rob > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [freezer] Proposing Saad Zaher for freezer core
+1. Saad Zaher is a Top Gear engineer : ) On Thu, Apr 14, 2016 at 4:20 PM, Mathieu, Pierre-Arthur < pierre-arthur.math...@hpe.com> wrote: > Hello, > > I would like to propose that we make Saad Zaher (szaher) core on freezer. > He has been a highly valuable developper for the past few month, mainly > working on integrating oslo component into freezer. > He has also been helping a lot with feature testing. > > His work can be found here: [1] > > Unless there is a disagreement I plan to make Saad core by the end of the > week. > > Thanks > - Pierre > > [1] https://review.openstack.org/#/q/owner:%22Saad+Zaher%22 > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Proposal: Separate design summits from OpenStack conferences
On Mon, Feb 8, 2016 at 11:19 AM, Jay Pipes wrote: > On 02/08/2016 10:29 AM, Sean Dague wrote: > >> On 02/08/2016 10:07 AM, Thierry Carrez wrote: >> >>> Brian Curtin wrote: >>> On Sun, Feb 7, 2016 at 3:07 PM, Jay Pipes wrote: > I would love to see the OpenStack contributor community take back the > design > summit to its original format and purpose and decouple it from the > OpenStack > Summit's conference portion. > > I believe the design summits should be organized by the OpenStack > contributor community, not the OpenStack Foundation and its marketing > and > event planning staff. > As someone who spent years organizing PyCon as a volunteer from the Python community, with four of those years in a row taking about 8 solid months of pre-conference effort, not to mention the on-site effort to run a volunteer conference of that size [0]...I would suggest even longer and harder thought before stretching a community like this even more thinly. Things should change, but probably not the "who's doing the work" aspect. >>> >>> Beyond stretching out the community, we would end up with the same >>> problem we are trying to solve. Most of the cross-project folks that >>> would end up organizing the event would be too busy organizing the event >>> to be able to fully participate in it. >>> >> >> Right, this is a super key point. Even just organizing and running local >> user groups, I know how much time is spent making sure the whole thing >> seems effortless to attendees, and they can just focus on content. >> >> Even look at the recently run Nova midcycle, with 40ish folks, it still >> required some substantial logistics to pull off. The HPE team did a >> great job with that. But it definitely required real time and effort. >> > > Agreed. > > The Foundation has done an amazing job of making everyone think this is >> easy (I know how much it is not). Without their efforts organizing these >> events, eliminating the distractions of wandering in a strange city to >> find lunch, having a network, projectors, access to facilities, >> appropriate sized spaces, double checking all those things will really >> actually be there, chasing after folks when they are not, handling the >> myriad of other unforseen issues that you never have to see we would >> not be nearly as productive at the design summits. >> > > I understand this. I ran the MySQL Users Conference and Expo for 2 years. > I realize the amount of effort it takes to organize a 2500+ person event. > It's essentially a full-time job. > > I suppose I should have used a different wording. What I really think > should happen is that a *separate* team should handle organizing the > developer-focused working events than the main team that does the marketing > event. I recognize that it's a lot of work and that asking the "community" > to just handle the working event organization will lead to undue burden on > certain cross-project folks. > > However, here are a couple things that do *not* need to be done by a > separate team that handles working event organization: > > 1) Vendor and sponsorship stuff > 2) A call for speakers and reviewing thousands of submissions (this is > self-organized by each project's contributor team for the working events) > 3) Determining keynote slots and wrangling C-level speakers -- or any > speaker wrangling at all > 4) "Check-in" and registration stands > 5) Dealing with schwag, giveaways, parties, and other superfluous stuff > > So, yes, while it's a lot of work, it's not the same kind of work as the > marketing event staff. > > So while I agree it's worth considering whether the Mega Conference and >> Design Summit should continue to be collocated and on the same time >> table, I think the idea that the Design Summit, at even only 500 >> attendees, could/should be run without the Foundation is just folly >> based on a lack of understanding for what it takes to do events at that >> scale. >> > > For the record, I *do* understand what it takes to do events at that scale. > > > And massively underestimates the effort and skill the Foundation > >> has at making our events run as smoothly as they do. >> > > I wasn't saying anything about the effort and skill the Foundation expends > on making the marketing events run smoothly. > > I am pushing for a return to *working* events for developers. > > -jay > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > Reworded like that , with no additional burden on the engineers, and without taking off the fan part of it, it makes a lots of sense. So are you proposing to do an engineering hardcore Design only summit, less expensive, like it was in the old days, some
Re: [openstack-dev] [all][tc] Proposal: Separate design summits from OpenStack conferences
On Mon, Feb 8, 2016 at 6:26 AM, Thierry Carrez wrote: > Daniel P. Berrange wrote: > >> [...] >> I really agree with everything you say, except for the bit about the >> community doing organization - I think its fine to let function event >> staff continue with the burden of planning, as long as their goals are >> directed by the community needs. >> > > Exactly. > > I might suggest that we could be a bit more radical with the developer >> event and decouple the timing from the release cycle. The design summits >> are portrayed as events where we plan the next 6 months of work, but the >> release has already been open for a good 2-3 or more weeks before we meet >> in the design summit. This always makes the first month of each >> development >> cycle pretty inefficient as decisions are needlessly postponed until the >> summit. The bulk of specs approval then doesn't happen until after the >> summit, leaving even less time until feature freeze to get the work done. >> > > I agree that the developer event happens too late in the cycle (3 weeks > after final release, 5 weeks after RC1 where most people switch to next > cycle, and 8 weeks after FF, where we start thinking about the next cycle). > That said, I still think the dev event should be "coupled" with the cycles. > It just needs to happen earlier. > > -- > Thierry Carrez (ttx) > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > The OpenStack Summit is a great thing as it is now. It creates big momentum, it's a strong motivator for the engineers (as enjoy our time there) and the Companies are happy too with the business related side. I see it also as the most successful Team building activity, Community and Company wide. For Companies, the costs to send engineers to the Summit or to a dedicated Design event are exactly the same. Besides, many Companies send US based employees only to the US Summit, and EU based only to the other side. The OpenStack Summit is probably the most advanced and successful OpenSource event, if you take out of it the engineering side, it won't be the same. I think, the issue here is that we need to have a better and more productive way to work together. Probably the motivation behind a separate design summit and also this discussion is focused to improve that, as we see that face to face is effective. Maybe this is the limitation we need to resolve, rather than changing an amazing event. Thanks, Fausto __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack
Hi Preston, Thank you. You saw Fabrizio in Vancouver, I'm Fausto, but it's allright, : P The challenge is interesting. If we want to build a dedicated backup API service (which is always what we wanted to do), probably we need to: - Place out of Nova and Cinder the backup features, as it wouldn't make much sense to me to have a Backup service and also have backups managed independently by Nova and Cinder. That said, I'm not a big fan of the following: - Interacting with the hypervisors and the volumes directly without passing through the Nova and Cinder API. - Adding any additional workload on the compute nodes or block storage nodes. - Computing incremental, compression, encryption is expensive. Have many simultaneous process doing that may lead to bad behaviours on core services. Some open questions: - Are we going to implement in Nova and Cinder, the features we need from the hypervisor for the backups i.e.: - http://wiki.qemu.org/Features/Snapshots2 - http://wiki.qemu.org/Features/Livebackup - Are we going to talk directly with the hypervisors without passing through the services API? - Are we going to provide a backup service using the feature from Nova and Cinder API currently implemented, and improve it over time as long as more related advanced features are available? - No other service touch directly volumes and hypervisors for reasons. We would need to make a diamond case to get an exception for backups. - There's any requirement from vendors and any of them will allocate resources to make a backup service, as a Team? My (flexible) thoughts are: - The feature is needed and is brilliant. - We should probably implement the newest feature provided by the hypervisor in Nova and export them from the Nova API. - Create a plugin that is integrated with Freezer to leverage that new features. - Same apply for Cinder. - The VMs and Volumes backup feature is already available by Nova, Cinder and Freezer. It needs to be improved for sure a lot, but do we need to create a new project for a feature that needs to be improved, rather than work with the existing Teams? - No one wants to block others, Sam proposed solution is indeed remarkable, but this is OpenStack, we work in Teams, why we cannot do that and be less fragmented. Thanks, Fausto On Mon, Feb 1, 2016 at 9:23 PM, Preston L. Bannister wrote: > Hi Fausto, > > To be clear, I am not in any way critical of Freezer and the folk putting > in work. (Please, I want to be *entirely* clear on this point! Also, saw > your presentation in Vancouver.) > > That said, Freezer is a bit of a Swiss-Army-Knife set of combined backup > functions. Sometimes it is better to focus on a single aspect (or few). The > new features landing in QEMU present an opportunity. A project focused > solely on that opportunity, to work through initial set of issues, makes a > lot of sense to me. (Something like how KVM forked QEMU for a time, to > build faster x86 emulation.) > > I do not see these as competing projects, and more as cooperative. The > Ekko work should be able to plug into Freezer, cleanly. > > Aspects of the problem, as I see: > >1. Extracting efficient instance backups from OpenStack (via new QEMU >function). >2. Storing backups in efficient form (general implementation, and >vendor-specific supersets). >3. Offering an OpenStack backup-service API, with core and >vendor-specific extensions. > > > Vendors (like my employer, EMC) might be somewhat opinionated about (2), > and for reason. :) > > The huge missing piece is (1), and a focused project seems to make a lot > of sense. > > As to (3), that looks like a good topic for further discussion. :) > > > My $.02. > > > > > On Sat, Jan 30, 2016 at 5:36 PM, Fausto Marzi > wrote: > >> Hi Preston, >> No need to apologize. They are aspect of the same problem. >> However, VMs backup is one of the many aspects that we are approaching >> here, such as: >> >> - VM backups >> - Volumes backups >> - Specific applications consistent data backup (i.e. MySQL, Mongo, file >> system, etc) >> - Provide capabilities to restore data even if keystone and swift are not >> available >> - Upload data during backup to multiple media storage in parallel >> - Web Interface >> - Provide capability to synchronize backups for sharded data on multiple >> nodes >> - Encryption >> - File based incremental >> - Block based incremental >> - Tenant related data backup and restore >> - Multi platform OS support (i.e. Linux, BSD, OSX, Windows, iOS, Android, >> etc) >> - Everything is upstreamed. >> >> This loo
Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack
Hi Preston, No need to apologize. They are aspect of the same problem. However, VMs backup is one of the many aspects that we are approaching here, such as: - VM backups - Volumes backups - Specific applications consistent data backup (i.e. MySQL, Mongo, file system, etc) - Provide capabilities to restore data even if keystone and swift are not available - Upload data during backup to multiple media storage in parallel - Web Interface - Provide capability to synchronize backups for sharded data on multiple nodes - Encryption - File based incremental - Block based incremental - Tenant related data backup and restore - Multi platform OS support (i.e. Linux, BSD, OSX, Windows, iOS, Android, etc) - Everything is upstreamed. This looks like a list of features... and actually it is. Block based and some multi platform OS aside, all the mentioned features are provided to the date. Most of them are available since Kilo. I agree with the multi API, room for vendors and to provide different approaches, but please let me say something (*not referring specifically to you or Sam or anyone*) All the time people say you have to do this and that, but the facts are that at the end of the day, always the same 6 engineers (not even full time) are working on it since 2 years, investing professional and personal time on it. We try to be open, to accept everybody (even the most arrogant), to implement features for whoever needs it, but the facts are that the only Companies that invested on it are HP, a bit Ericsson and Orange (apologize if I forgot anyone). We never said no to anyone about anything, never focused only to a single Company influence, never blocked a thing... and never will. Wouldn't be better to join efforts if companies need a backup solution and have their own requirements implemented by a common public Team, rather then start creating many tools to solve the same set of problems? How can ever competition benefit this? How can ever fragmenting projects help to provide a better solution? I'm sorry, but unless the TC or many people from the Community, tell us to do something different (in that case we'll do it straight away), we'll keep doing what we are doing, focusing on delivering what we think is the most advanced solution, according the resources and time we have. We need to understand that here the most important thing is to work in Team, to provide great tools to the Community, rather then thinking to be PTL or maintain independence just for the sake of it or focusing only on what's the best for a single Company. If this vision is not shared, then, unfortunately, good luck competing, while if the vision is shared... let's do together unprecedented things. Many thanks, Fausto On Sun, Jan 31, 2016 at 1:01 AM, Preston L. Bannister wrote: > Seems to me there are three threads here. > > The Freezer folk were given a task, and did the best possible to support > backup given what OpenStack allowed. To date, OpenStack is simply not very > good at supporting backup as a service. (Apologies to the Freezer folk if I > misinterpreted.) > > The patches (finally) landing in QEMU in support of incremental backup > could be the basis for efficient backup services in OpenStack. This is all > fairly high risk, in the short term. The bits that landed in QEMU 2.4 may > not be sufficient (there are more QEMU patches trying to land). When put > into production, we may find faults. For use in OpenStack, we may need > changes in libvirt, and/or in Nova. (Or *maybe* not if usage for backup > proves orthogonal.) The only way to work out the prior is to start. The > timeline could be months or years. > > There is a need for a common API for backup as a service in the cloud. > Something more than imitating AWS. Allow some room for vendors with > differing approach. > > I see the above as not competing, but aspects of the same problem. > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack
What motivates me every day and every night, is to provide the most advanced solution for a set a problems to solve, Open Source and in OpenStack. What motivates me is to work with brilliant people like minded, capable of doing great things working together. It is not the competition and It is not the PTL or Core label that gives you that. In OpenStack most of the time, new project born because there's a group of people in a company that needs to provide a solution for a set of problems, and the same people/company allocate resources, time and efforts. Now if every company start his own project, we are likely to going to have many individual tools to provide a similar set of solutions. Is this what we want? I hope not. Starting new projects is a tremendously hard work and with limited resources most of the time. So why not join, work as a community, provide a remarkable service/tool and secure the result for who decide to use OpenStack. We have to do the effort of trying hard to work together, smooth differences, improve others rather than block them. Honestly as a PTL I try to do that, every single bloody day, dealing with managers, companies, customers, engineers, code, bugs and community. However I do not think a PTL should be elected 2 consecutive times, for official projects, because it's a great experience that anyone that deserve it, should learn from. But this is just my opinion. That said, what we are trying to do, with Sam (that is showing an open attitude and constructive approach), is to understand if the solution make sense itself, if we can work together as a Team and if it make sense to include it in Freezer. If not, it is crystal clear, that no one in the world, can stop a very committed and capable person to do what he/she wants to do. Even less in the Open Source and here. We probably need to avoid fragmentation and at the same time no limiting new ideas, but potentiate them. Happy to hear your thoughts on this. Thanks, Fausto On Thu, Jan 28, 2016 at 8:55 PM, Ian Wells wrote: > On 27 January 2016 at 11:06, Flavio Percoco wrote: > >> FWIW, the current governance model does not prevent competition. That's >> not to >> be understood as we encourage it but rather than there could be services >> with >> some level of overlap that are still worth being separate. >> > > There should always be the possibility to compete; it's always possible > that rethinking an idea produces a better implementation of the same API. > But we don't separate API from implementation in Openstack - the 'XXX API' > cannot easily be divorced from the project containing the implementation > when the definition of the 'XXX API' is 'the API implemented by the XXX > code'. We should separate them - the API is the only thing that a tenant > will ever actually care about, not the implementation choice behind it. > > What Jay is referring to is that regardless the projects do similar >> things, the >> same or totally different things, we should strive to have different >> APIs. The >> API shouldn't overlap in terms of endpoints and the way they are exposed. >> > > And for this, perhaps we should have an API namespace registry, so that > when two groups implement the same endpoints they have to implement the > same API? I think Jay's point is that we muddy the waters here by having > confusingly similar-but-different APIs. > > [The counterargument is that service discovery usually determines what API > you're getting, so that if two services claim to be different service types > in Keystone then they are *not* the same API and should be allowed free > reign of their URI namespace, but I see that's not working for us.] > > And, coming back to the original point, if Freezer and Ekko both implement > backups, and they can come to an agreement on what 'a backup' is and an API > definition for it, that means that they could exist as independent projects > with independent codebases that both implement /backup - but, importantly, > in a consistent way that doesn't confuse app developers. That will only > work if the API definition stands separate from the projects, though. > > -- > Ian. > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack
Hi Sam, After our conversation, I have few questions and consideration about Ekko, mainly on how it works et similar. Also to make available to the community our discussions: - In understand you are placing a backup-agent on the compute node and execute actions interacting directly with the hypervisor. I’m thinking that while Ekko execute this actions, the Nova service have no visibility whatsoever of this. I do not think is a good idea to execute actions directly on the hypervisor without interacting with the Nova API. - On your assumptions, you said that Nova snapshots creation generate a VM downtime. I don’t think the assumption is correct, at least in Kilo, Liberty and Mitaka. The only downtime you may have related to the snapshot, is when you merge back the snapshot to the original root image, and this is not our case here. - How the restore would work? If you do a restore of the VM and the record of that VM instance is not available in the Nova DB (i.e. restoring a VM on a newly installed Openstack cloud, or in another region, or after a vm has beed destroyed)what would happen? How do you manage the consistency of the data between Nova DB and VM status - If you execute a backup of the VM image file without executing a backup of the related VM metadata information (in the shortest time frame as possible) there are chances the backup can be inconsistent. - How the restore would happen if on that moment Keystone or Swift is not available? - Does the backup that Ekko execute, generates bootable image? If not, the image is not usable and the restore process will take longer to execute the steps to make the image bootable. - I do not see any advantage in Ekko over using Nova API to snapshot -> Generate an image -> upload to Glance -> upload to Swift. - The Ekko approach is limited to Nova, KVM QEMU, having a qemu-agent running on the VM. I think the scope is probably a bit limited. This is more a feature than a tool itself, but the problem is being solved I think more efficiently already. - By executing all the actions related to backup (i.e. compression, incremental computation, upload, I/O and segmented upload to Swift) Ekko is adding a significant load to the Compute Nodes. All the work is done on the hypervisor and not taken into account by ceilometer (or similar), so for example not billable. I do not think this is a good idea as distributing the load over multiple components helps OpenStack to scale and by leveraging the existing API you integrated better with existing tools. - There’s no documentation whatsoever provided with Ekko. I had to read the source code, have conversations directly with you and invest significant time on it. I think provide some documentation is helpful, as the doc link in the openstack/ekko repo return 404 Not Found. Please let me know what your thoughts are on this. Thanks, Fausto On Wed, Jan 27, 2016 at 1:55 PM, Sam Yaple wrote: > On Wed, Jan 27, 2016 at 7:06 PM, Flavio Percoco wrote: >> >> FWIW, the current governance model does not prevent competition. That's >> not to >> be understood as we encourage it but rather than there could be services >> with >> some level of overlap that are still worth being separate. >> >> What Jay is referring to is that regardless the projects do similar >> things, the >> same or totally different things, we should strive to have different >> APIs. The >> API shouldn't overlap in terms of endpoints and the way they are exposed. >> >> With all that said, I'd like to encourage collaboration over competition >> and I'm >> sure both teams will find a way to make this work. >> >> Cheers, >> Flavio > > > And to come full circle on this thread, I will point out once again there > is no competition between Ekko and Freezer at this time. Freezer is > file-level backup where Ekko is block-level backup. Anyone familiar with > backups knows these are drastically different types of backups. Those using > block-level backups typically won't be using file-level backups and > vice-versa. That said, even if there is no convergence of Freezer and Ekko > they can still live side-by-side without any conflict at all. > > As of now, Ekko and Freezer teams have started a dialogue and we will > continue to collaborate rather than compete in every way that is reasonable > for both projects. > > Sam Yaple > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack
Hi Jay, Dean, totally agree, with Sam, we'll make sure there will not be any overlap. Sam, Thanks for your openness. I'd like to have a conversation about it. When you say "current direction of Ekko will add many components" do you have any reference, plan or road map where we can see that direction and components more in detail? Very soon we'll starting working also on the block based incremental, that will be overlapping... if Ekko would fit and we do not have to re implement it, that'd be fantastic, and not overlapping =). I see many points we can converge. >From our perspective, we try to focus more on what can be converged rather then what cannot. Converging is something that allow us to move forward faster, delivering something better to the Community, that's the only motivation. Let's talk on what we can do, I'm totally confident we'll find a way to do something useful for everybody : ), beside we matured some experience dealing positively with the OS community, that will benefit you, believe me : ) Sounds good? I'll set up a public meeting next week to discuss this. Many thanks, Fausto On Tue, Jan 26, 2016 at 11:03 AM, Sam Yaple wrote: > Didn't hit the mailing list with the last reply. Forwarding to a wider > audience than just Dean > -- Forwarded message -- > From: "Sam Yaple" > Date: Jan 26, 2016 12:00 PM > Subject: Re: [openstack-dev] Announcing Ekko -- Scalable block-based > backup for OpenStack > To: "Dean Troyer" > Cc: > > > On Jan 26, 2016 11:42 AM, "Dean Troyer" wrote: > > > > On Tue, Jan 26, 2016 at 9:28 AM, Sam Yaple wrote: > >> > >> On Tue, Jan 26, 2016 at 10:15 AM, Jay Pipes wrote: > >>> > >>> My personal request is that the two contributor communities do > everything in their power to ensure that the REST API endpoints are not > overlapping. The last thing we need is to have two APIs for performing > backups that are virtually identical to each other. > >>> > >> > >> The way I see this situation is the same as asking Ekko to integrate > with cinder-backup because they perform backups that are "virtually > identical" to each other. They aren't actually related at all, other than > perhaps an API > > > > > > But you see this is exactly where they are directly related to everyone > who is not a developer of the back-end services. Everything that wants to > do a volume backup (users, other services, etc) should not have multiple > choices to perform that backup, irregardless of how that action is > implemented. > > You skipped over the important part. "Actual implementation and end > results are wildly different." They are not the same even if the API call > is named similar, as I originally stated. > > > Perhaps this is a problem whose time has come to address? > > Absolutely! I would love to not have to worry about building and > maintaining an API. That said, Ekko isn't here to solve that issue. > > > I would hate to see us keep duplicating user-facing stuff for the sake > of back-end developer convenience > > I agree about not duplicating user-facing things. But it is a bit more > than "back-end developer convenience". I am both an operator and a > developer, so I can speak from both of those points of view when I say I do > not want to be forced to deploy services that I won't use for a feature > unrelated to them. For sale of example, requiring Ceilometer to use Cinder > would be awful. Those wanting to use Ekko may have no interest in using > Freezer and vice versa. Forcing unrelated processes and services for the > sake of one API is not something I agree with. > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack
Hi Sam, My opinion would be to converge, so to have Ekko features exported from the freezer-api and horizon web interface. Also the freezer-scheduler can be integrated, that would enable Ekko to execute backup syncronized over multiple nodes. By all mean, this does not mean you have to, it's just how I feel about it. We are totally open, so please let us know if there's any interest from your side. Thanks, Fausto Sent from my iPhone > On 25 Jan 2016, at 08:58, Sam Yaple wrote: > >> On Mon, Jan 25, 2016 at 8:45 AM, Thierry Carrez >> wrote: >> Sam Yaple wrote: >>> We would like to introduce you to a new community-driven OpenStack >>> project called Ekko. >>> >>> The aim of Ekko is to provide incremental block-level backup and restore >>> of Nova instances. We see backups as a key area that is missing in >>> OpenStack. One issue that has previously prevented backups in OpenStack >>> is the scalability of the storage backend. Object-storage is the answer >>> to this scalability problem, but with block-based backups you often see >>> large files that require POSIX operations to perform retention and >>> deletions. These operations are not able to be performed in the >>> traditional way in object storage, which has prevented leveraging >>> object-storage to its full potential. With Ekko we can solve this issue >>> allowing us to use storage that can scale with OpenStack. >> >> Hi! >> >> How does Ekko compare to / differ from Freezer, which is an official >> OpenStack project targeted to the same problem space ? I suspect this is >> more low-level ? Is there some potential for convergence between the two >> projects ? > > Hello Thierry, > > These are good questions. The biggest difference you already caught onto, > Ekko is more low-level. Freezer is targeted at the filesystem and specific > applications (like databases) directly. > > There are only two places with overlapping goals that I know of*. The first > is backup of a Cinder volume which is a future goal of Ekko and something > Freezer can currently do for LVM backed Cinder. The second is backup of a > nova instance. This isn't something freezer does directly, instead it > leverages nova-snapshot which is very disruptive to the instance and will > cause downtime for said instance. The current pursuit of Ekko is _live_ > incremental block-level backup of an nova instance and in that regard there > is no overlap with Freezer or any other project for that matter. > > To answer the question of convergence between Ekko and Freezer, I would say > it's possible. That being said both projects are addressing different > problems in different ways. As discussed above, there is little overlap > between the two projects and the areas where there is overlap of goals have > drastically different implementations. I could put together a list of Ekko vs > Freezer, but I think that would be comparing apples to oranges. To state this > in terms of compatibility, Ekko and Freezer can run side-by-side without > interfering with each other in anyway. > > *Disclaimer: I am no expert on Freezer, I may be wrong in some statements and > am open to correction. > > Sam Yaple > >> -- >> Thierry Carrez (ttx) >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [freezer][agent][bug] Trying Freezerc with no success
Hi Deklan, Following up our conversation on restoring from swift, yes there's a bug, you are right, I've reproduced it. It can be reproduced when python-swiftclient 2.7.0 is installed on the system. In Liberty and Mitaka the requirement is python-swiftclient>=2.2.0. The issue should be solved by doing the following change in requirements.txt -python-swiftclient>=2.2.0 +python-swiftclient>=2.2.0,<=2.6.0 I've opened the following freezer bug: - https://bugs.launchpad.net/freezer/+bug/1537364 The patches to should solve the issue are: Mitaka: - https://review.openstack.org/#/c/271701 Liberty (cherry-pick): - https://review.openstack.org/#/c/271702/ Many thanks for your effort, you are most welcome in the Freezer Team. Fausto __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [freezer] Meetings in pause state until 7th of Jan
Hi All, We'll be mostly off until the 7th of Jan. If you need anything urgent please drop me an email. I want to take this opportunity to thank the Freezer Team for the amazing job, effort and sacrifices faced during the 2015. You have all my respect, as you already know : ) Have the most fantastic holidays. Thanks, Fausto __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
