Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Sat, Feb 6, 2016 at 5:12 PM, Duncan Thomas wrote: > Actually, keeping track of changed blocks on cinder volumes would make the > cinder incremental backup substantially more efficient... Something could > push them into cinder at detach time, and an api call for cinder to pull > them at live b

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

Actually, keeping track of changed blocks on cinder volumes would make the cinder incremental backup substantially more efficient... Something could push them into cinder at detach time, and an api call for cinder to pull them at live backup time, and cinder backup can do the rest... Not sure of th

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Sat, Feb 6, 2016 at 3:00 PM, Jeremy Stanley wrote: > On 2016-02-05 16:38:19 + (+), Sam Yaple wrote: > > I always forget to qualify that statement don't I? Nova does not > > have a mechanism for _incremental_ backups. Nor does Nova have > > compression or encryption because AFAIK that a

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 2016-02-05 16:38:19 + (+), Sam Yaple wrote: > I always forget to qualify that statement don't I? Nova does not > have a mechanism for _incremental_ backups. Nor does Nova have > compression or encryption because AFAIK that api only creates a > snapshot. I would also point out again that

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 02/05/2016 11:38 AM, Sam Yaple wrote: On Fri, Feb 5, 2016 at 3:31 PM, Jay Pipes mailto:jaypi...@gmail.com>> wrote: On 02/05/2016 09:58 AM, Sam Yaple wrote: Since Nova has no backup mechanism this is clearly a gap and that was the issue Ekko wants to solve. N

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Fri, Feb 5, 2016 at 3:31 PM, Jay Pipes wrote: > On 02/05/2016 09:58 AM, Sam Yaple wrote: > >> Since Nova has no backup mechanism this is clearly a gap and that was the >> issue >> Ekko wants to solve. >> > > Nova has had backups for a long time: > > http://developer.openstack.org/api-ref-compu

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 02/05/2016 09:58 AM, Sam Yaple wrote: Since Nova has no backup mechanism this is clearly a gap and that was the issue Ekko wants to solve. Nova has had backups for a long time: http://developer.openstack.org/api-ref-compute-v2.1.html#createBackup Best, -jay ___

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Thu, Feb 4, 2016 at 2:23 PM, gordon chung wrote: > > > On 03/02/2016 10:38 AM, Sam Yaple wrote: > > On Wed, Feb 3, 2016 at 2:52 PM, Jeremy Stanley < > fu...@yuggoth.org> wrote: > >> On 2016-02-03 14:32:36 + (+), Sam Yaple wrote: >> [...] >> > Luckily, digging into it it appears cinder

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 03/02/2016 10:38 AM, Sam Yaple wrote: On Wed, Feb 3, 2016 at 2:52 PM, Jeremy Stanley <fu...@yuggoth.org> wrote: On 2016-02-03 14:32:36 + (+), Sam Yaple wrote: [...] > Luckily, digging into it it appears cinder already has all the > i

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Wed, Feb 3, 2016 at 4:53 PM, Preston L. Bannister wrote: > On Wed, Feb 3, 2016 at 6:32 AM, Sam Yaple wrote: > >> [snip] >> > Full backups are costly in terms of IO, storage, bandwidth and time. A >> full backup being required in a backup plan is a big problem for backups >> when we talk about

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Wed, Feb 3, 2016 at 6:32 AM, Sam Yaple wrote: > [snip] > Full backups are costly in terms of IO, storage, bandwidth and time. A full > backup being required in a backup plan is a big problem for backups when we > talk about volumes that are terabytes large. > As an incidental note... You hav

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Wed, Feb 3, 2016 at 3:58 PM, Duncan Thomas wrote: > On 3 February 2016 at 17:52, Sam Yaple wrote: > > >> This is a very similiar method to what Ekko is doing. The json mapping in >> Ekko is a manifest file which is a sqlite database. The major difference I >> see is Ekko is doing backup trees

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

tOn 3 February 2016 at 17:52, Sam Yaple wrote: > This is a very similiar method to what Ekko is doing. The json mapping in > Ekko is a manifest file which is a sqlite database. The major difference I > see is Ekko is doing backup trees. If you launch 1000 instances from the > same glance image,

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Wed, Feb 3, 2016 at 3:36 PM, Duncan Thomas wrote: > On 3 February 2016 at 17:27, Sam Yaple wrote: > > >> >> And here we get to the meat of the matter. Squashing backups is awful in >> object storage. It requires you to pull both backups, merge them, then >> reupload. This also has the downsid

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Wed, Feb 3, 2016 at 2:52 PM, Jeremy Stanley wrote: > On 2016-02-03 14:32:36 + (+), Sam Yaple wrote: > [...] > > Luckily, digging into it it appears cinder already has all the > > infrastructure in place to handle what we had talked about in a > > separate email thread Duncan. It is ver

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 3 February 2016 at 17:27, Sam Yaple wrote: > > And here we get to the meat of the matter. Squashing backups is awful in > object storage. It requires you to pull both backups, merge them, then > reupload. This also has the downside of casting doubt on a backup since you > are now modifying da

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Wed, Feb 3, 2016 at 2:37 PM, Duncan Thomas wrote: > > > On 3 February 2016 at 16:32, Sam Yaple wrote: > >> >> Looking into it, however, shows Cinder has no mechanism to delete backups >> in the middle of a chain since you use dependent backups (please correct me >> if I am wrong here). This m

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 2016-02-03 14:32:36 + (+), Sam Yaple wrote: [...] > Luckily, digging into it it appears cinder already has all the > infrastructure in place to handle what we had talked about in a > separate email thread Duncan. It is very possible Ekko can > leverage the existing features to do it's ba

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Wed, Feb 3, 2016 at 1:41 PM, Duncan Thomas wrote: > On 2 February 2016 at 02:28, Sam Yaple wrote: > >> >> I disagree with this statement strongly as I have stated before. Nova has >> snapshots. Cinder has snapshots (though they do say cinder-backup). Freezer >> wraps Nova and Cinder. Snapshot

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 3 February 2016 at 16:32, Sam Yaple wrote: > > Looking into it, however, shows Cinder has no mechanism to delete backups > in the middle of a chain since you use dependent backups (please correct me > if I am wrong here). This means after a number of incremental backups you > _must_ take anoth

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 2 February 2016 at 02:28, Sam Yaple wrote: > > I disagree with this statement strongly as I have stated before. Nova has > snapshots. Cinder has snapshots (though they do say cinder-backup). Freezer > wraps Nova and Cinder. Snapshots are not backups. They are certainly not > _incremental_ back

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Tue, Feb 2, 2016 at 11:31 PM, Sean McGinnis wrote: > On Tue, Feb 02, 2016 at 02:04:44PM -0800, Preston L. Bannister wrote: > > I submitted for a presentation on "State of the Art for in-Cloud backup > of > > high-value Applications". Notion is to give context for the folk who need > > this sor

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Tue, Feb 02, 2016 at 02:04:44PM -0800, Preston L. Bannister wrote: > On a side note, of the folk with interest in this thread, how many are > going to the Austin OpenStack conference? Would you be interested in > presenting as a panel? > > I submitted for a presentation on "State of the Art for

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On a side note, of the folk with interest in this thread, how many are going to the Austin OpenStack conference? Would you be interested in presenting as a panel? I submitted for a presentation on "State of the Art for in-Cloud backup of high-value Applications". Notion is to give context for the

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Feb 2, 2016 7:41 AM, "Preston L. Bannister" wrote: > > Oh, for the other folk reading, in QEMU you want to look at: > > http://wiki.qemu.org/Features/IncrementalBackup > > The above page looks to be current. The QEMU wiki seems to have a number of stale pages that describe proposed function tha

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

Oh, for the other folk reading, in QEMU you want to look at: http://wiki.qemu.org/Features/IncrementalBackup The above page looks to be current. The QEMU wiki seems to have a number of stale pages that describe proposed function that was abandoned / never implemented. Originally, I ended up readi

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

To be clear, I work for EMC, and we are building a backup product for OpenStack (which at this point is very far along). The primary lack is a good means to efficiently extract changed-block information from OpenStack. About a year ago I worked through the entire Nova/Cinder/libvirt/QEMU stack, to

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Mon, Feb 1, 2016 at 10:32 PM, Fausto Marzi wrote: > Hi Preston, > Thank you. You saw Fabrizio in Vancouver, I'm Fausto, but it's allright, : > P > > The challenge is interesting. If we want to build a dedicated backup API > service (which is always what we wanted to do), probably we need to: >

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Mon, Feb 1, 2016 at 8:23 PM, Preston L. Bannister wrote: > Hi Fausto, > > To be clear, I am not in any way critical of Freezer and the folk putting > in work. (Please, I want to be *entirely* clear on this point! Also, saw > your presentation in Vancouver.) > > That said, Freezer is a bit of a

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

Hi Preston, Thank you. You saw Fabrizio in Vancouver, I'm Fausto, but it's allright, : P The challenge is interesting. If we want to build a dedicated backup API service (which is always what we wanted to do), probably we need to: - Place out of Nova and Cinder the backup features, as it woul

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

Hi Fausto, To be clear, I am not in any way critical of Freezer and the folk putting in work. (Please, I want to be *entirely* clear on this point! Also, saw your presentation in Vancouver.) That said, Freezer is a bit of a Swiss-Army-Knife set of combined backup functions. Sometimes it is better

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

Fausto Marzi wrote: [...] That said, what we are trying to do, with Sam (that is showing an open attitude and constructive approach), is to understand if the solution make sense itself, if we can work together as a Team and if it make sense to include it in Freezer. If not, it is crystal clear, t

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

Hi Preston, No need to apologize. They are aspect of the same problem. However, VMs backup is one of the many aspects that we are approaching here, such as: - VM backups - Volumes backups - Specific applications consistent data backup (i.e. MySQL, Mongo, file system, etc) - Provide capabilities to

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

Seems to me there are three threads here. The Freezer folk were given a task, and did the best possible to support backup given what OpenStack allowed. To date, OpenStack is simply not very good at supporting backup as a service. (Apologies to the Freezer folk if I misinterpreted.) The patches (f

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

What motivates me every day and every night, is to provide the most advanced solution for a set a problems to solve, Open Source and in OpenStack. What motivates me is to work with brilliant people like minded, capable of doing great things working together. It is not the competition and It is not

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 27 January 2016 at 11:06, Flavio Percoco wrote: > FWIW, the current governance model does not prevent competition. That's > not to > be understood as we encourage it but rather than there could be services > with > some level of overlap that are still worth being separate. > There should alwa

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 28/01/2016 1:03 AM, Steven Dake (stdake) wrote: > Choice is one of life's fundamental elements that make life interesting. > Let the Operators choose what they wish based upon technical merits rather > then "who was first to publish a project". yes, it's nice when competing entities exists in

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 01/28/2016 06:03 AM, Steven Dake (stdake) wrote: The TC set precedent in this first competition-induced review which is worth a good read for other projects thinking of competing with existing projects of which there are already plenty in OpenStack.. https://review.openstack.org/206789 Stev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 1/27/16, 12:06 PM, "Flavio Percoco" wrote: >On 27/01/16 12:16 -0500, Emilien Macchi wrote: >> >> >>On 01/27/2016 10:51 AM, Jay Pipes wrote: >>> On 01/27/2016 12:53 PM, gordon chung wrote: > It makes for a crappy user experience. Crappier than the crappy user > experience that OpenSta

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Wed, Jan 27, 2016 at 8:19 PM, Fausto Marzi wrote: > Hi Sam, > > After our conversation, I have few questions and consideration about Ekko, > mainly on how it works et similar. Also to make available to the community > our discussions: > > - In understand you are placing a backup-agent

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

Hi Sam, After our conversation, I have few questions and consideration about Ekko, mainly on how it works et similar. Also to make available to the community our discussions: - In understand you are placing a backup-agent on the compute node and execute actions interacting directly with

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Wed, Jan 27, 2016 at 7:06 PM, Flavio Percoco wrote: > > FWIW, the current governance model does not prevent competition. That's > not to > be understood as we encourage it but rather than there could be services > with > some level of overlap that are still worth being separate. > > What Jay is

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 27/01/16 12:16 -0500, Emilien Macchi wrote: On 01/27/2016 10:51 AM, Jay Pipes wrote: On 01/27/2016 12:53 PM, gordon chung wrote: It makes for a crappy user experience. Crappier than the crappy user experience that OpenStack API users already have because we have done a crappy job shepherdi

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 01/27/2016 10:51 AM, Jay Pipes wrote: > On 01/27/2016 12:53 PM, gordon chung wrote: >>> It makes for a crappy user experience. Crappier than the crappy user >>> experience that OpenStack API users already have because we have done a >>> crappy job shepherding projects in order to make sure the

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 27/01/2016 10:51 AM, Jay Pipes wrote: > On 01/27/2016 12:53 PM, gordon chung wrote: >>> It makes for a crappy user experience. Crappier than the crappy user >>> experience that OpenStack API users already have because we have done a >>> crappy job shepherding projects in order to make sure the

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 01/27/2016 12:53 PM, gordon chung wrote: It makes for a crappy user experience. Crappier than the crappy user experience that OpenStack API users already have because we have done a crappy job shepherding projects in order to make sure there isn't overlap between their APIs (yes, Ceilometer an

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

> It makes for a crappy user experience. Crappier than the crappy user > experience that OpenStack API users already have because we have done a > crappy job shepherding projects in order to make sure there isn't > overlap between their APIs (yes, Ceilometer and Monasca, I'm looking > directly at y

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Tue, Jan 26, 2016 at 9:57 PM, Jay Pipes wrote: > > I am not suggesting you "share an API" at all. I am requesting that if you > have a RESTful API planned for your "backup", then you do not use the same > RESTful API resource endpoint names that Freezer does. Because if you do, > then users of

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 01/26/2016 03:28 PM, Sam Yaple wrote: On Tue, Jan 26, 2016 at 10:15 AM, Jay Pipes mailto:jaypi...@gmail.com>> wrote: On 01/26/2016 02:47 AM, Sam Yaple wrote: Hello Fausto, I am happy to have a conversation about this with you and the Freezer team. I have

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

6 at 11:03 AM, Sam Yaple wrote: > Didn't hit the mailing list with the last reply. Forwarding to a wider > audience than just Dean > -- Forwarded message -- > From: "Sam Yaple" > Date: Jan 26, 2016 12:00 PM > Subject: Re: [openstack-dev] Announcing

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

Didn't hit the mailing list with the last reply. Forwarding to a wider audience than just Dean -- Forwarded message -- From: "Sam Yaple" Date: Jan 26, 2016 12:00 PM Subject: Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack To: &q

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Tue, Jan 26, 2016 at 9:28 AM, Sam Yaple wrote: > On Tue, Jan 26, 2016 at 10:15 AM, Jay Pipes wrote: > >> My personal request is that the two contributor communities do everything >> in their power to ensure that the REST API endpoints are not overlapping. >> The last thing we need is to have

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Tue, Jan 26, 2016 at 10:15 AM, Jay Pipes wrote: > On 01/26/2016 02:47 AM, Sam Yaple wrote: > >> Hello Fausto, >> >> I am happy to have a conversation about this with you and the Freezer >> team. I have a feeling the current direction of Ekko will add many >> components that will not be needed

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On 01/26/2016 02:47 AM, Sam Yaple wrote: Hello Fausto, I am happy to have a conversation about this with you and the Freezer team. I have a feeling the current direction of Ekko will add many components that will not be needed for Freezer and vice-versa. Nevertheless, I am all about community!

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

Hello Fausto, I am happy to have a conversation about this with you and the Freezer team. I have a feeling the current direction of Ekko will add many components that will not be needed for Freezer and vice-versa. Nevertheless, I am all about community! Sam Yaple On Tue, Jan 26, 2016 at 2:20 AM,

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

Hi Sam, My opinion would be to converge, so to have Ekko features exported from the freezer-api and horizon web interface. Also the freezer-scheduler can be integrated, that would enable Ekko to execute backup syncronized over multiple nodes. By all mean, this does not mean you have to, it's ju

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

On Mon, Jan 25, 2016 at 8:45 AM, Thierry Carrez wrote: > Sam Yaple wrote: > >> We would like to introduce you to a new community-driven OpenStack >> project called Ekko. >> >> The aim of Ekko is to provide incremental block-level backup and restore >> of Nova instances. We see backups as a key ar

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

Sam Yaple wrote: We would like to introduce you to a new community-driven OpenStack project called Ekko. The aim of Ekko is to provide incremental block-level backup and restore of Nova instances. We see backups as a key area that is missing in OpenStack. One issue that has previously prevented

[openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

Dear community, We would like to introduce you to a new community-driven OpenStack project called Ekko. The aim of Ekko is to provide incremental block-level backup and restore of Nova instances. We see backups as a key area that is missing in OpenStack. One issue that has previously prevented ba