Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Jeremy Stanley]

On 2016-02-05 16:38:19 + (+), Sam Yaple wrote:
> I always forget to qualify that statement don't I? Nova does not
> have a mechanism for _incremental_ backups. Nor does Nova have
> compression or encryption because AFAIK that api only creates a
> snapshot. I would also point out again that snapshots != backups,
> at least not for those who care about backups.

And just to be clear, you assert that the Nova team would reject
extending their existing backup implementation to support this, so
the only real solution is to make another project.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Duncan Thomas]

Actually, keeping track of changed blocks on cinder volumes would make the
cinder incremental backup substantially more efficient... Something could
push them into cinder at detach time, and an api call for cinder to pull
them at live backup time, and cinder backup can do the rest... Not sure of
the none-cinder bits of the architecture, but certainly an interesting
idea. In the event something goes wrong, cinder can assume the while device
has changed or fall back to the current mechanism, so it is back compatible
from a tenant point of view...
On 6 Feb 2016 17:56, "Sam Yaple"  wrote:

> On Sat, Feb 6, 2016 at 3:00 PM, Jeremy Stanley  wrote:
>
>> On 2016-02-05 16:38:19 + (+), Sam Yaple wrote:
>> > I always forget to qualify that statement don't I? Nova does not
>> > have a mechanism for _incremental_ backups. Nor does Nova have
>> > compression or encryption because AFAIK that api only creates a
>> > snapshot. I would also point out again that snapshots != backups,
>> > at least not for those who care about backups.
>>
>> And just to be clear, you assert that the Nova team would reject
>> extending their existing backup implementation to support this, so
>> the only real solution is to make another project.
>>
>
> I don't know if Nova would reject it or not, but as discussed it could be
> extended to Cinder. Should Nova ever backup Cinder volumes? Additionally,
> why don't we combine networking into Nova? Or images? Or volumes? What I do
> assert is that we have done alot of work to strip out components from Nova,
> backups don't seem like a good candidate to shove into Nova.
>
> Luckily since Ekko and Nova (just like Ekko and Freezer) don't have any
> conflicting operations should Ekko be built separate and merged into Nova
> it would be fairly painless process since there are no overlapping services.
>
> Integration with Nova where Nova controls the hypervisor and Ekko requests
> operations through the Nova api before doing the backup is another
> question, and that is reasonable in my opinion. This is likely an issue
> that can be addressed down the road rather than at this moment, though.
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Sat, Feb 6, 2016 at 5:12 PM, Duncan Thomas 
wrote:

> Actually, keeping track of changed blocks on cinder volumes would make the
> cinder incremental backup substantially more efficient... Something could
> push them into cinder at detach time, and an api call for cinder to pull
> them at live backup time, and cinder backup can do the rest... Not sure of
> the none-cinder bits of the architecture, but certainly an interesting
> idea. In the event something goes wrong, cinder can assume the while device
> has changed or fall back to the current mechanism, so it is back compatible
> from a tenant point of view...
>
The mechanism that nova could call right now has no libvirt counterpart. So
with Ekko I am talking to the qemu process through the monitor socket to
initiate these commands. QEMU spits out the data I need to backup, I am as
of now unsure if the changed block bitmap itself can be extracted or if it
is all an internal tracking (I haven't looked into this).

I can see a future where, though nova-api's, both Cinder and Ekko can
perform backups without the raw data having to traverse through nova
itself, just the metadata such as changed-blocks and what not. This would
be my preferred solution rather than have Nova itself run the backup and
transfer of data to the backing storage (not to mention scheduling and
retention). Additionally it would mean that neither Ekko nor Cinder would
need to talk to the hypervisors directly and can still utilize that
low-level info.


> On 6 Feb 2016 17:56, "Sam Yaple"  wrote:
>
>> On Sat, Feb 6, 2016 at 3:00 PM, Jeremy Stanley  wrote:
>>
>>> On 2016-02-05 16:38:19 + (+), Sam Yaple wrote:
>>> > I always forget to qualify that statement don't I? Nova does not
>>> > have a mechanism for _incremental_ backups. Nor does Nova have
>>> > compression or encryption because AFAIK that api only creates a
>>> > snapshot. I would also point out again that snapshots != backups,
>>> > at least not for those who care about backups.
>>>
>>> And just to be clear, you assert that the Nova team would reject
>>> extending their existing backup implementation to support this, so
>>> the only real solution is to make another project.
>>>
>>
>> I don't know if Nova would reject it or not, but as discussed it could be
>> extended to Cinder. Should Nova ever backup Cinder volumes? Additionally,
>> why don't we combine networking into Nova? Or images? Or volumes? What I do
>> assert is that we have done alot of work to strip out components from Nova,
>> backups don't seem like a good candidate to shove into Nova.
>>
>> Luckily since Ekko and Nova (just like Ekko and Freezer) don't have any
>> conflicting operations should Ekko be built separate and merged into Nova
>> it would be fairly painless process since there are no overlapping services.
>>
>> Integration with Nova where Nova controls the hypervisor and Ekko
>> requests operations through the Nova api before doing the backup is another
>> question, and that is reasonable in my opinion. This is likely an issue
>> that can be addressed down the road rather than at this moment, though.
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Jay Pipes]

On 02/05/2016 09:58 AM, Sam Yaple wrote:

Since Nova has no backup mechanism this is clearly a gap and that was the issue
Ekko wants to solve.


Nova has had backups for a long time:

http://developer.openstack.org/api-ref-compute-v2.1.html#createBackup

Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Fri, Feb 5, 2016 at 3:31 PM, Jay Pipes  wrote:

> On 02/05/2016 09:58 AM, Sam Yaple wrote:
>
>> Since Nova has no backup mechanism this is clearly a gap and that was the
>> issue
>> Ekko wants to solve.
>>
>
> Nova has had backups for a long time:
>
> http://developer.openstack.org/api-ref-compute-v2.1.html#createBackup
>
> I always forget to qualify that statement don't I? Nova does not have a
mechanism for _incremental_ backups. Nor does Nova have compression or
encryption because AFAIK that api only creates a snapshot. I would also
point out again that snapshots != backups, at least not for those who care
about backups.

Sam Yaple
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Jay Pipes]

On 02/05/2016 11:38 AM, Sam Yaple wrote:

On Fri, Feb 5, 2016 at 3:31 PM, Jay Pipes > wrote:

On 02/05/2016 09:58 AM, Sam Yaple wrote:

Since Nova has no backup mechanism this is clearly a gap and
that was the issue
Ekko wants to solve.


Nova has had backups for a long time:

http://developer.openstack.org/api-ref-compute-v2.1.html#createBackup

I always forget to qualify that statement don't I?


No worries :)

> Nova does not have a

mechanism for _incremental_ backups. Nor does Nova have compression or
encryption because AFAIK that api only creates a snapshot. I would also
point out again that snapshots != backups, at least not for those who
care about backups.


Understood.

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Thu, Feb 4, 2016 at 2:23 PM, gordon chung  wrote:

>
>
> On 03/02/2016 10:38 AM, Sam Yaple wrote:
>
> On Wed, Feb 3, 2016 at 2:52 PM, Jeremy Stanley < 
> fu...@yuggoth.org> wrote:
>
>> On 2016-02-03 14:32:36 + (+), Sam Yaple wrote:
>> [...]
>> > Luckily, digging into it it appears cinder already has all the
>> > infrastructure in place to handle what we had talked about in a
>> > separate email thread Duncan. It is very possible Ekko can
>> > leverage the existing features to do it's backup with no change
>> > from Cinder.
>> [...]
>>
>> If Cinder's backup facilities already do most of
>> what you want from it and there's only a little bit of development
>> work required to add the missing feature, why jump to implementing
>> this feature in a completely separate project instead rather than
>> improving Cinder's existing solution so that people who have been
>> using that can benefit directly?
>>
>
> Backing up Cinder was never the initial goal, just a potential feature on
> the roadmap. Nova is the main goal.
>
> i'll extend fungi's question, are the backup framework/mechanisms common
> whether it be Nova or Cinder or anything else? or are they unique but only
> grouped together as a service because they backup something. it seems the
> problem is we've imagined the service as tackling a horizontal issue when
> really it is just a vertical story that appears across many silos.
>
>
The framework would not be common even between Ekko and Cinder, and there
is no Nova "backup". The main featureset that will be utilized is CBT
(changed-block-tracking) which is needed for efficient incremental backups.
Otherwise the entire block device must be read each time a backup is
performed. The design of Ekko solves the horizontal issue of scaling
backups, there is also this vertical integration that _can_ be done to make
the whole experience more pleasant and reliable. Since Nova has no backup
mechanism this is clearly a gap and that was the issue Ekko wants to solve.
Also backuping up Cinder volumes is on the roadmap, but has not been a
stated priority.

Sam Yaple


> cheers,
>
> --
> gord
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[gordon chung]

On 03/02/2016 10:38 AM, Sam Yaple wrote:
On Wed, Feb 3, 2016 at 2:52 PM, Jeremy Stanley 
<fu...@yuggoth.org> wrote:
On 2016-02-03 14:32:36 + (+), Sam Yaple wrote:
[...]
> Luckily, digging into it it appears cinder already has all the
> infrastructure in place to handle what we had talked about in a
> separate email thread Duncan. It is very possible Ekko can
> leverage the existing features to do it's backup with no change
> from Cinder.
[...]

If Cinder's backup facilities already do most of
what you want from it and there's only a little bit of development
work required to add the missing feature, why jump to implementing
this feature in a completely separate project instead rather than
improving Cinder's existing solution so that people who have been
using that can benefit directly?

Backing up Cinder was never the initial goal, just a potential feature on the 
roadmap. Nova is the main goal.

i'll extend fungi's question, are the backup framework/mechanisms common 
whether it be Nova or Cinder or anything else? or are they unique but only 
grouped together as a service because they backup something. it seems the 
problem is we've imagined the service as tackling a horizontal issue when 
really it is just a vertical story that appears across many silos.

cheers,

--
gord
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Duncan Thomas]

On 2 February 2016 at 02:28, Sam Yaple  wrote:

>
> I disagree with this statement strongly as I have stated before. Nova has
> snapshots. Cinder has snapshots (though they do say cinder-backup). Freezer
> wraps Nova and Cinder. Snapshots are not backups. They are certainly not
> _incremental_ backups. They can have neither compression, nor encryption.
> With this in mind, Freezer does not have this "feature" at all. Its not
> that it needs improvement, it simply does not exist in Freezer. So a
> separate project dedicated to that one goal is not unreasonable. The real
> question is whether it is practical to merge Freezer and Ekko, and this is
> the question Ekko and the Freezer team are attempting to answer.
>

You're misinformed of the cinder feature set there - cinder has both
snapshots (usually fast COW thing on the same storage backend) and backups
(copy to a different storage backend, usually swift but might be
NFS/ceph/TSM) - the backups support incremental and compression. Separate
encryption to the volume encryption is not yet supported or implemented,
merely because nobody has written it yet. There's also live backup
(internally via a snapshot) merged last cycle.

I can see a place for other backup solutions, I just want to make the
existing ones clear.

-- 
Duncan Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Duncan Thomas]

On 3 February 2016 at 16:32, Sam Yaple  wrote:

>
> Looking into it, however, shows Cinder has no mechanism to delete backups
> in the middle of a chain since you use dependent backups (please correct me
> if I am wrong here). This means after a number of incremental backups you
> _must_ take another full to ensure the chain doesn't get to long. That is a
> problem Ekko is purposing to solve as well. Full backups are costly in
> terms of IO, storage, bandwidth and time. A full backup being required in a
> backup plan is a big problem for backups when we talk about volumes that
> are terabytes large.
>

You're right that this is an issue currently. Cinder actually has enough
info in theory to be able to trivially squash backups to be able to break
the chain, it's only a bit of metadata ref counting and juggling, however
nobody has yet written the code.


> Luckily, digging into it it appears cinder already has all the
> infrastructure in place to handle what we had talked about in a separate
> email thread Duncan. It is very possible Ekko can leverage the existing
> features to do it's backup with no change from Cinder. This isn't the
> initial priority for Ekko though, but it is good information to have. Thank
> you for your comments!
>


Always interested in better ways to solve backup.


-- 
Duncan Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Duncan Thomas]

On 3 February 2016 at 17:27, Sam Yaple  wrote:


>
> And here we get to the meat of the matter. Squashing backups is awful in
> object storage. It requires you to pull both backups, merge them, then
> reupload. This also has the downside of casting doubt on a backup since you
> are now modifying data after it has been backed up (though that doubt is
> lessened with proper checksuming/hashing which cinder does it looks like).
> This is the issue Ekko can solve (and has solved over the past 2 years).
> Ekko can do this "squashing" in a non-traditional way, without ever
> modifying content or merging anything. With deletions only. This means we
> do not have to pull two backups, merge, and reupload to delete a backup
> from the chain.
>

I'm sure we've lost most of the audience by this point, but I might as well
reply here as anywhere else...

In the cinder backup case, since the backup is chunked in object store, all
that is required is to reference count the chunks that are required for the
backups you want to keep, get rid of the rest, and re-upload the (very
small) json mapping file. You can either upload over the old json, or
create a new one. Either way, the bulk data does not need to be touched.



-- 
-- 
Duncan Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Wed, Feb 3, 2016 at 1:41 PM, Duncan Thomas 
wrote:

> On 2 February 2016 at 02:28, Sam Yaple  wrote:
>
>>
>> I disagree with this statement strongly as I have stated before. Nova has
>> snapshots. Cinder has snapshots (though they do say cinder-backup). Freezer
>> wraps Nova and Cinder. Snapshots are not backups. They are certainly not
>> _incremental_ backups. They can have neither compression, nor encryption.
>> With this in mind, Freezer does not have this "feature" at all. Its not
>> that it needs improvement, it simply does not exist in Freezer. So a
>> separate project dedicated to that one goal is not unreasonable. The real
>> question is whether it is practical to merge Freezer and Ekko, and this is
>> the question Ekko and the Freezer team are attempting to answer.
>>
>
> You're misinformed of the cinder feature set there - cinder has both
> snapshots (usually fast COW thing on the same storage backend) and backups
> (copy to a different storage backend, usually swift but might be
> NFS/ceph/TSM) - the backups support incremental and compression. Separate
> encryption to the volume encryption is not yet supported or implemented,
> merely because nobody has written it yet. There's also live backup
> (internally via a snapshot) merged last cycle.
>
> You are right Duncan. I was working on outdated information that Cinder
does not have incremental backups. I apologize for the misstep there, we
haven't started on the Cinder planning yet so I haven't looked into it in
great detail.

Looking into it, however, shows Cinder has no mechanism to delete backups
in the middle of a chain since you use dependent backups (please correct me
if I am wrong here). This means after a number of incremental backups you
_must_ take another full to ensure the chain doesn't get to long. That is a
problem Ekko is purposing to solve as well. Full backups are costly in
terms of IO, storage, bandwidth and time. A full backup being required in a
backup plan is a big problem for backups when we talk about volumes that
are terabytes large.

Luckily, digging into it it appears cinder already has all the
infrastructure in place to handle what we had talked about in a separate
email thread Duncan. It is very possible Ekko can leverage the existing
features to do it's backup with no change from Cinder. This isn't the
initial priority for Ekko though, but it is good information to have. Thank
you for your comments!
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Jeremy Stanley]

On 2016-02-03 14:32:36 + (+), Sam Yaple wrote:
[...]
> Luckily, digging into it it appears cinder already has all the
> infrastructure in place to handle what we had talked about in a
> separate email thread Duncan. It is very possible Ekko can
> leverage the existing features to do it's backup with no change
> from Cinder.
[...]

If Cinder's backup facilities already do most of
what you want from it and there's only a little bit of development
work required to add the missing feature, why jump to implementing
this feature in a completely separate project instead rather than
improving Cinder's existing solution so that people who have been
using that can benefit directly?
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Wed, Feb 3, 2016 at 2:37 PM, Duncan Thomas 
wrote:

>
>
> On 3 February 2016 at 16:32, Sam Yaple  wrote:
>
>>
>> Looking into it, however, shows Cinder has no mechanism to delete backups
>> in the middle of a chain since you use dependent backups (please correct me
>> if I am wrong here). This means after a number of incremental backups you
>> _must_ take another full to ensure the chain doesn't get to long. That is a
>> problem Ekko is purposing to solve as well. Full backups are costly in
>> terms of IO, storage, bandwidth and time. A full backup being required in a
>> backup plan is a big problem for backups when we talk about volumes that
>> are terabytes large.
>>
>
> You're right that this is an issue currently. Cinder actually has enough
> info in theory to be able to trivially squash backups to be able to break
> the chain, it's only a bit of metadata ref counting and juggling, however
> nobody has yet written the code.
>
>
And here we get to the meat of the matter. Squashing backups is awful in
object storage. It requires you to pull both backups, merge them, then
reupload. This also has the downside of casting doubt on a backup since you
are now modifying data after it has been backed up (though that doubt is
lessened with proper checksuming/hashing which cinder does it looks like).
This is the issue Ekko can solve (and has solved over the past 2 years).
Ekko can do this "squashing" in a non-traditional way, without ever
modifying content or merging anything. With deletions only. This means we
do not have to pull two backups, merge, and reupload to delete a backup
from the chain.


> Luckily, digging into it it appears cinder already has all the
>> infrastructure in place to handle what we had talked about in a separate
>> email thread Duncan. It is very possible Ekko can leverage the existing
>> features to do it's backup with no change from Cinder. This isn't the
>> initial priority for Ekko though, but it is good information to have. Thank
>> you for your comments!
>>
>
>
> Always interested in better ways to solve backup.
>

Thats the plan!

>
> --
> Duncan Thomas
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Duncan Thomas]

tOn 3 February 2016 at 17:52, Sam Yaple  wrote:


> This is a very similiar method to what Ekko is doing. The json mapping in
> Ekko is a manifest file which is a sqlite database. The major difference I
> see is Ekko is doing backup trees. If you launch 1000 instances from the
> same glance image, you don't need 1000 fulls, you need 1 full and 1000
> incrementals. Doing that means you save a ton of space, time, bandwidth,
> IO, but it also means n number of backups can reference the same chunk of
> data and it makes deletion of that data much harder than you describe in
> Cinder. When restoring a backup, you don't _need_ a new full, you need to
> start your backups based on the last restore point and the same point about
> saving applies. It also means that Ekko can provide "backups can scale with
> OpenStack" in that sense. Your backups will only ever be your changed data.
>
> I recognize that isn't probably a huge concern for Cinder, with volumes
> typically being just unique data and not duplicate data, but with nova I
> would argue _most_ instances in an OpenStack deployment will be based on
> the same small subset of images and thats alot of duplicate data to
> consider backing up especially at scale.
>
>

So this sounds great. If your backup formats are similar enough, it is
worth considering putting a backup export function in that spits out a
cinder-backup compatible JSON file (it's a dead simple format) and perhaps
an import for the same. That would allow cinder backup and Ekko to exchange
data where desired. I'm not sure if this is possible, but I'd certainly
suggest looking at it.

Thanks for keeping the dialog open, it has definitely been useful.


-- 
Duncan Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Preston L. Bannister]

On Wed, Feb 3, 2016 at 6:32 AM, Sam Yaple  wrote:

> [snip]
>
Full backups are costly in terms of IO, storage, bandwidth and time. A full
> backup being required in a backup plan is a big problem for backups when we
> talk about volumes that are terabytes large.
>

As an incidental note...

You have to collect full backups, periodically. To do otherwise
assumes *absolutely
no failures* anywhere in the entire software/hardware stack -- ever -- and
no failures in storage over time. (Which collectively is a tad optimistic,
at scale.) Whether due to a rare software bug, a marginal piece of
hardware, or a stray cosmic ray - an occasional bad block will slip through.

More exactly, you need some means of doing occasional full end-to-end
verification of stored backups. Periodic full backups are one
safeguard. How you go about performing full verification, and how often is
a subject for design and optimization. This is where things get a *bit*
more complex. :)

Or you just accept a higher error rate. (How high depends on the
implementation.)

And "Yes", multi-terabyte volumes *are* a challenge.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Wed, Feb 3, 2016 at 3:36 PM, Duncan Thomas 
wrote:

> On 3 February 2016 at 17:27, Sam Yaple  wrote:
>
>
>>
>> And here we get to the meat of the matter. Squashing backups is awful in
>> object storage. It requires you to pull both backups, merge them, then
>> reupload. This also has the downside of casting doubt on a backup since you
>> are now modifying data after it has been backed up (though that doubt is
>> lessened with proper checksuming/hashing which cinder does it looks like).
>> This is the issue Ekko can solve (and has solved over the past 2 years).
>> Ekko can do this "squashing" in a non-traditional way, without ever
>> modifying content or merging anything. With deletions only. This means we
>> do not have to pull two backups, merge, and reupload to delete a backup
>> from the chain.
>>
>
> I'm sure we've lost most of the audience by this point, but I might as
> well reply here as anywhere else...
>

That's ok. We are talking and thats important for featuresets that people
don't even know they want!

>
> In the cinder backup case, since the backup is chunked in object store,
> all that is required is to reference count the chunks that are required for
> the backups you want to keep, get rid of the rest, and re-upload the (very
> small) json mapping file. You can either upload over the old json, or
> create a new one. Either way, the bulk data does not need to be touched.
>

This is a very similiar method to what Ekko is doing. The json mapping in
Ekko is a manifest file which is a sqlite database. The major difference I
see is Ekko is doing backup trees. If you launch 1000 instances from the
same glance image, you don't need 1000 fulls, you need 1 full and 1000
incrementals. Doing that means you save a ton of space, time, bandwidth,
IO, but it also means n number of backups can reference the same chunk of
data and it makes deletion of that data much harder than you describe in
Cinder. When restoring a backup, you don't _need_ a new full, you need to
start your backups based on the last restore point and the same point about
saving applies. It also means that Ekko can provide "backups can scale with
OpenStack" in that sense. Your backups will only ever be your changed data.

I recognize that isn't probably a huge concern for Cinder, with volumes
typically being just unique data and not duplicate data, but with nova I
would argue _most_ instances in an OpenStack deployment will be based on
the same small subset of images and thats alot of duplicate data to
consider backing up especially at scale.

I will have to understand a bit more about cinder-backup before I approach
that subject with Ekko (which right now is on the newton roadmap). What you
have told me absolutely justifies the cinder-backup name (rather than
cinder-snapshot) so thank you for correcting me on that point!


>
>
>
> --
> --
> Duncan Thomas
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Wed, Feb 3, 2016 at 2:52 PM, Jeremy Stanley  wrote:

> On 2016-02-03 14:32:36 + (+), Sam Yaple wrote:
> [...]
> > Luckily, digging into it it appears cinder already has all the
> > infrastructure in place to handle what we had talked about in a
> > separate email thread Duncan. It is very possible Ekko can
> > leverage the existing features to do it's backup with no change
> > from Cinder.
> [...]
>
> If Cinder's backup facilities already do most of
> what you want from it and there's only a little bit of development
> work required to add the missing feature, why jump to implementing
> this feature in a completely separate project instead rather than
> improving Cinder's existing solution so that people who have been
> using that can benefit directly?
>

Backing up Cinder was never the initial goal, just a potential feature on
the roadmap. Nova is the main goal.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Wed, Feb 3, 2016 at 3:58 PM, Duncan Thomas 
wrote:

> On 3 February 2016 at 17:52, Sam Yaple  wrote:
>
>
>> This is a very similiar method to what Ekko is doing. The json mapping in
>> Ekko is a manifest file which is a sqlite database. The major difference I
>> see is Ekko is doing backup trees. If you launch 1000 instances from the
>> same glance image, you don't need 1000 fulls, you need 1 full and 1000
>> incrementals. Doing that means you save a ton of space, time, bandwidth,
>> IO, but it also means n number of backups can reference the same chunk of
>> data and it makes deletion of that data much harder than you describe in
>> Cinder. When restoring a backup, you don't _need_ a new full, you need to
>> start your backups based on the last restore point and the same point about
>> saving applies. It also means that Ekko can provide "backups can scale with
>> OpenStack" in that sense. Your backups will only ever be your changed data.
>>
>> I recognize that isn't probably a huge concern for Cinder, with volumes
>> typically being just unique data and not duplicate data, but with nova I
>> would argue _most_ instances in an OpenStack deployment will be based on
>> the same small subset of images and thats alot of duplicate data to
>> consider backing up especially at scale.
>>
>>
>
> So this sounds great. If your backup formats are similar enough, it is
> worth considering putting a backup export function in that spits out a
> cinder-backup compatible JSON file (it's a dead simple format) and perhaps
> an import for the same. That would allow cinder backup and Ekko to exchange
> data where desired. I'm not sure if this is possible, but I'd certainly
> suggest looking at it.
>
> This is potentially possible. The issue I see would be around
compression/encryption of the different chunks (in ekko we refer to them as
segments). But we will probably be able to work this out in time.


> Thanks for keeping the dialog open, it has definitely been useful.
>
> I have enjoyed the exchange as well. I am a big fan of open-source and
community.

>
> --
> Duncan Thomas
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Wed, Feb 3, 2016 at 4:53 PM, Preston L. Bannister 
wrote:

> On Wed, Feb 3, 2016 at 6:32 AM, Sam Yaple  wrote:
>
>> [snip]
>>
> Full backups are costly in terms of IO, storage, bandwidth and time. A
>> full backup being required in a backup plan is a big problem for backups
>> when we talk about volumes that are terabytes large.
>>
>
> As an incidental note...
>
> You have to collect full backups, periodically. To do otherwise assumes 
> *absolutely
> no failures* anywhere in the entire software/hardware stack -- ever --
> and no failures in storage over time. (Which collectively is a tad
> optimistic, at scale.) Whether due to a rare software bug, a marginal piece
> of hardware, or a stray cosmic ray - an occasional bad block will slip
> through.
>

A new full can be triggered at any time should there be concern of a
problem. (see my next point)

>
> More exactly, you need some means of doing occasional full end-to-end
> verification of stored backups. Periodic full backups are one
> safeguard. How you go about performing full verification, and how often is
> a subject for design and optimization. This is where things get a *bit*
> more complex. :)
>

Yes an end-to-end verification of the backup would be easy to implement,
but costly to run. But thats more on the user to decided those things. With
a proper scheduler this is less an issue for Ekko, and more a backup policy
issue.

>
> Or you just accept a higher error rate. (How high depends on the
> implementation.)
>

And its not a full loss, its just not a 100% valid backup. Luckily youve
only lost a single segment (a few thousand sectors) chances are the
critical stuff you want isn't there. That data can still be recovered.
And object-storage with replication makes it very, very hard to loss data
when properly maintained (look at S3 and the data its lost over time). We
have checksum/hash verification in place already so the underlying data
must be valid or we don't restore. But your points are well received.

>
> And "Yes", multi-terabyte volumes *are* a challenge.
>

And increasingly common...
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Preston L. Bannister]

On a side note, of the folk with interest in this thread, how many are
going to the Austin OpenStack conference? Would you be interested in
presenting as a panel?

I submitted for a presentation on "State of the Art for in-Cloud backup of
high-value Applications". Notion is to give context for the folk who need
this sort of backup. Something about where we have been, where we are, and
what might become possible. I think it would be great to pull in folk from
Freezer and Ekko. Jay Pipes seems to like to weigh in on the topic, and
could represent Nova? Will gladly add as speakers interested folk! (Of
course, the odds of winning a slot are pretty low, but worth a shot.)

Any folk who expect to be in Austin, and are interested?
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sean McGinnis]

On Tue, Feb 02, 2016 at 02:04:44PM -0800, Preston L. Bannister wrote:
> On a side note, of the folk with interest in this thread, how many are
> going to the Austin OpenStack conference? Would you be interested in
> presenting as a panel?
> 
> I submitted for a presentation on "State of the Art for in-Cloud backup of
> high-value Applications". Notion is to give context for the folk who need
> this sort of backup. Something about where we have been, where we are, and
> what might become possible. I think it would be great to pull in folk from
> Freezer and Ekko. Jay Pipes seems to like to weigh in on the topic, and
> could represent Nova? Will gladly add as speakers interested folk! (Of
> course, the odds of winning a slot are pretty low, but worth a shot.)

I'd love to see a presentation on Ekko, Freezer, Smaug, as well as the
built in capabilities such as Cinder backup.

It would be interesting to see an overview of all the existing and planned
capabilities for backup. It might also be a good way to gain interest in
one or more of these projects, and possibly some operator feedback on
what would be interesting and desired in a final solution.

> 
> Any folk who expect to be in Austin, and are interested?

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Tue, Feb 2, 2016 at 11:31 PM, Sean McGinnis 
wrote:

> On Tue, Feb 02, 2016 at 02:04:44PM -0800, Preston L. Bannister wrote:
> > I submitted for a presentation on "State of the Art for in-Cloud backup
> of
> > high-value Applications". Notion is to give context for the folk who need
> > this sort of backup. Something about where we have been, where we are,
> and
> > what might become possible. I think it would be great to pull in folk
> from
> > Freezer and Ekko. Jay Pipes seems to like to weigh in on the topic, and
> > could represent Nova? Will gladly add as speakers interested folk! (Of
> > course, the odds of winning a slot are pretty low, but worth a shot.)
>
> I'd love to see a presentation on Ekko, Freezer, Smaug, as well as the
> built in capabilities such as Cinder backup.
>

I submitted an Ekko talk "The 'B' Word -- Backups in OpenStack". Title
seems all inclusive, but in reality I am just talking about the block-based
side of backups. I am co-presenting with another Ekko dev and we do have a
brief slot in our outline for explaining Ekko's place in the OpenStack
ecosystem and its difference from nova-snapshot or cinder-backup.

Sam Yaple
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Preston L. Bannister]

To be clear, I work for EMC, and we are building a backup product for
OpenStack (which at this point is very far along). The primary lack is a
good means to efficiently extract changed-block information from OpenStack.
About a year ago I worked through the entire Nova/Cinder/libvirt/QEMU
stack, to see what was possible. The changes to QEMU (which have been
in-flight since 2011) looked most promising, but when they would land was
unclear. They are starting to land. This is big news. :)

That is not the end of the problem. Unless the QEMU folk are perfect, there
are likely bugs to be found when the code is put into production. (With
more exercise, the sooner any problems can be identified and addressed.)
OpenStack uses libvirt to talk to QEMU, and libvirt is a fairly thick
abstraction. Likely there will want to be adjustments to libvirt. Bypassing
Nova and chatting with libvirt directly is a bit suspect (but may be
needed). There might be adjustments needed in Nova.

To offer suggestions...

Ekko is an *opinionated* approach to backup. This is not the only way to
solve the problem. I happen very much like the approach, but as a *specific
*approach, it probably does not belong in Cinder or Nova. (I believe it was
Jay who offered a similar argument about backup more generally.)

(Keep in mind QEMU is not the only hypervisor supported by Nova, if the
majority of use. Would you want to attempt a design that works for all
hypervisors? I would not!  ...at least at this point. Also, last I checked
the Cinder folk were a bit hung up on replication, as finding common
abstractions across storage was not easy. This problem looks similar.)

While wary of bypassing Nova/Cinder, my suggestion would to be rude in the
beginning, with every intent of becoming civil in the end.

Start by talking to libvirt directly. (The was a bypass mechanism in
libvirt that looked like it might be sufficient.) Break QEMU early, and get
it fixed. :)

When QEMU usage is working, talk to the libvirt folk about *proven* needs,
and what is needed to become civil.

When libvirt is updated (or not), talk to Nova folk about *proven* needs,
and what is needed to become civil. (Perhaps simply awareness, or a small
set of primitives.)

It might take quite a while for the latest QEMU and libvirt to ripple
through into OpenStack distributions. Getting any fixes into QEMU early (or
addressing discovered gaps in needed function) seems like a good thing.

All the above is a sufficiently ambitious project, just by itself. To my
mind, that justifies Ekko as a unique, focused project.






On Mon, Feb 1, 2016 at 4:28 PM, Sam Yaple  wrote:

> On Mon, Feb 1, 2016 at 10:32 PM, Fausto Marzi 
> wrote:
>
>> Hi Preston,
>> Thank you. You saw Fabrizio in Vancouver, I'm Fausto, but it's allright,
>> : P
>>
>> The challenge is interesting. If we want to build a dedicated backup API
>> service (which is always what we wanted to do), probably we need to:
>>
>>
>>- Place out of Nova and Cinder the backup features, as it wouldn't
>>make much sense to me to have a Backup service and also have backups
>>managed independently by Nova and Cinder.
>>
>>
>> That said, I'm not a big fan of the following:
>>
>>- Interacting with the hypervisors and the volumes directly without
>>passing through the Nova and Cinder API.
>>
>> Passing through the api will be a huge issue for extracting data due to
> the sheer volume of data needed (TB through the api is going to kill
> everything!)
>
>>
>>- Adding any additional workload on the compute nodes or block
>>storage nodes.
>>- Computing incremental, compression, encryption is expensive. Have
>>many simultaneous process doing that may lead  to bad behaviours on core
>>services.
>>
>> These are valid concerns, but the alternative is still shipping the raw
> data elsewhere to do this work, and that has its own issue in terms of
> bandwidth.
>
>>
>> My (flexible) thoughts are:
>>
>>- The feature is needed and is brilliant.
>>- We should probably implement the newest feature provided by the
>>hypervisor in Nova and export them from the Nova API.
>>- Create a plugin that is integrated with Freezer to leverage that
>>new features.
>>- Same apply for Cinder.
>>- The VMs and Volumes backup feature is already available by Nova,
>>Cinder and Freezer. It needs to be improved for sure a lot, but do we need
>>to create a new project for a feature that needs to be improved, rather
>>than work with the existing Teams?
>>
>> I disagree with this statement strongly as I have stated before. Nova has
> snapshots. Cinder has snapshots (though they do say cinder-backup). Freezer
> wraps Nova and Cinder. Snapshots are not backups. They are certainly not
> _incremental_ backups. They can have neither compression, nor encryption.
> With this in mind, Freezer does not have this "feature" at all. Its not
> that it needs improvement, it simply 

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Preston L. Bannister]

Oh, for the other folk reading, in QEMU you want to look at:

http://wiki.qemu.org/Features/IncrementalBackup

The above page looks to be current. The QEMU wiki seems to have a number of
stale pages that describe proposed function that was abandoned / never
implemented. Originally, I ended up reading the QEMU mailing list and
source code to figure out which bits were real. :)





On Tue, Feb 2, 2016 at 4:04 AM, Preston L. Bannister 
wrote:

> To be clear, I work for EMC, and we are building a backup product for
> OpenStack (which at this point is very far along). The primary lack is a
> good means to efficiently extract changed-block information from OpenStack.
> About a year ago I worked through the entire Nova/Cinder/libvirt/QEMU
> stack, to see what was possible. The changes to QEMU (which have been
> in-flight since 2011) looked most promising, but when they would land was
> unclear. They are starting to land. This is big news. :)
>
> That is not the end of the problem. Unless the QEMU folk are perfect,
> there are likely bugs to be found when the code is put into production.
> (With more exercise, the sooner any problems can be identified and
> addressed.) OpenStack uses libvirt to talk to QEMU, and libvirt is a fairly
> thick abstraction. Likely there will want to be adjustments to libvirt.
> Bypassing Nova and chatting with libvirt directly is a bit suspect (but may
> be needed). There might be adjustments needed in Nova.
>
> To offer suggestions...
>
> Ekko is an *opinionated* approach to backup. This is not the only way to
> solve the problem. I happen very much like the approach, but as a *specific
> *approach, it probably does not belong in Cinder or Nova. (I believe it
> was Jay who offered a similar argument about backup more generally.)
>
> (Keep in mind QEMU is not the only hypervisor supported by Nova, if the
> majority of use. Would you want to attempt a design that works for all
> hypervisors? I would not!  ...at least at this point. Also, last I checked
> the Cinder folk were a bit hung up on replication, as finding common
> abstractions across storage was not easy. This problem looks similar.)
>
> While wary of bypassing Nova/Cinder, my suggestion would to be rude in the
> beginning, with every intent of becoming civil in the end.
>
> Start by talking to libvirt directly. (The was a bypass mechanism in
> libvirt that looked like it might be sufficient.) Break QEMU early, and get
> it fixed. :)
>
> When QEMU usage is working, talk to the libvirt folk about *proven*
> needs, and what is needed to become civil.
>
> When libvirt is updated (or not), talk to Nova folk about *proven* needs,
> and what is needed to become civil. (Perhaps simply awareness, or a small
> set of primitives.)
>
> It might take quite a while for the latest QEMU and libvirt to ripple
> through into OpenStack distributions. Getting any fixes into QEMU early (or
> addressing discovered gaps in needed function) seems like a good thing.
>
> All the above is a sufficiently ambitious project, just by itself. To my
> mind, that justifies Ekko as a unique, focused project.
>
>
>
>
>
>
> On Mon, Feb 1, 2016 at 4:28 PM, Sam Yaple  wrote:
>
>> On Mon, Feb 1, 2016 at 10:32 PM, Fausto Marzi 
>> wrote:
>>
>>> Hi Preston,
>>> Thank you. You saw Fabrizio in Vancouver, I'm Fausto, but it's allright,
>>> : P
>>>
>>> The challenge is interesting. If we want to build a dedicated backup API
>>> service (which is always what we wanted to do), probably we need to:
>>>
>>>
>>>- Place out of Nova and Cinder the backup features, as it wouldn't
>>>make much sense to me to have a Backup service and also have backups
>>>managed independently by Nova and Cinder.
>>>
>>>
>>> That said, I'm not a big fan of the following:
>>>
>>>- Interacting with the hypervisors and the volumes directly without
>>>passing through the Nova and Cinder API.
>>>
>>> Passing through the api will be a huge issue for extracting data due to
>> the sheer volume of data needed (TB through the api is going to kill
>> everything!)
>>
>>>
>>>- Adding any additional workload on the compute nodes or block
>>>storage nodes.
>>>- Computing incremental, compression, encryption is expensive. Have
>>>many simultaneous process doing that may lead  to bad behaviours on core
>>>services.
>>>
>>> These are valid concerns, but the alternative is still shipping the raw
>> data elsewhere to do this work, and that has its own issue in terms of
>> bandwidth.
>>
>>>
>>> My (flexible) thoughts are:
>>>
>>>- The feature is needed and is brilliant.
>>>- We should probably implement the newest feature provided by the
>>>hypervisor in Nova and export them from the Nova API.
>>>- Create a plugin that is integrated with Freezer to leverage that
>>>new features.
>>>- Same apply for Cinder.
>>>- The VMs and Volumes backup feature is already available by Nova,
>>>

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Feb 2, 2016 7:41 AM, "Preston L. Bannister"  wrote:
>
> Oh, for the other folk reading, in QEMU you want to look at:
>
> http://wiki.qemu.org/Features/IncrementalBackup
>
> The above page looks to be current. The QEMU wiki seems to have a number
of stale pages that describe proposed function that was abandoned / never
implemented. Originally, I ended up reading the QEMU mailing list and
source code to figure out which bits were real. :)

Unfortunately the link you provided is old as well. Source code is the only
way to go at this point for valid information. But I have a working
structure for a backup using QMP directly if not manually at this point.

>
>
>
>
>
> On Tue, Feb 2, 2016 at 4:04 AM, Preston L. Bannister 
wrote:
>>
>> To be clear, I work for EMC, and we are building a backup product for
OpenStack (which at this point is very far along). The primary lack is a
good means to efficiently extract changed-block information from OpenStack.
About a year ago I worked through the entire Nova/Cinder/libvirt/QEMU
stack, to see what was possible. The changes to QEMU (which have been
in-flight since 2011) looked most promising, but when they would land was
unclear. They are starting to land. This is big news. :)
>>
>> That is not the end of the problem. Unless the QEMU folk are perfect,
there are likely bugs to be found when the code is put into production.
(With more exercise, the sooner any problems can be identified and
addressed.) OpenStack uses libvirt to talk to QEMU, and libvirt is a fairly
thick abstraction. Likely there will want to be adjustments to libvirt.
Bypassing Nova and chatting with libvirt directly is a bit suspect (but may
be needed). There might be adjustments needed in Nova.
>>
>> To offer suggestions...
>>
>> Ekko is an opinionated approach to backup. This is not the only way to
solve the problem. I happen very much like the approach, but as a specific
approach, it probably does not belong in Cinder or Nova. (I believe it was
Jay who offered a similar argument about backup more generally.)
>>
>> (Keep in mind QEMU is not the only hypervisor supported by Nova, if the
majority of use. Would you want to attempt a design that works for all
hypervisors? I would not!  ...at least at this point. Also, last I checked
the Cinder folk were a bit hung up on replication, as finding common
abstractions across storage was not easy. This problem looks similar.)

Actually, now that you mention it, the approach Ekko takes _does_ work
against all hypervisors that support changed-block-tracking. At this point
that includes QEMU, VMWare, and HyperV (no Xen).
>>
>> While wary of bypassing Nova/Cinder, my suggestion would to be rude in
the beginning, with every intent of becoming civil in the end.
>>
>> Start by talking to libvirt directly. (The was a bypass mechanism in
libvirt that looked like it might be sufficient.) Break QEMU early, and get
it fixed. :)
>>
>> When QEMU usage is working, talk to the libvirt folk about proven needs,
and what is needed to become civil.
>>
>> When libvirt is updated (or not), talk to Nova folk about proven needs,
and what is needed to become civil. (Perhaps simply awareness, or a small
set of primitives.)

It's like you are a mind reader! This is my exact thoughts on the approach
as well.
>>
>> It might take quite a while for the latest QEMU and libvirt to ripple
through into OpenStack distributions. Getting any fixes into QEMU early (or
addressing discovered gaps in needed function) seems like a good thing.
>>
>> All the above is a sufficiently ambitious project, just by itself. To my
mind, that justifies Ekko as a unique, focused project.
>>
Thank you for you input. This is the case in my mind as well, but if this
goal can be achieved _better_ with Freezer we absolutely should meld with
Freezer. That's the question we are trying to figure out.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Preston L. Bannister]

Hi Fausto,

To be clear, I am not in any way critical of Freezer and the folk putting
in work. (Please, I want to be *entirely* clear on this point! Also, saw
your presentation in Vancouver.)

That said, Freezer is a bit of a Swiss-Army-Knife set of combined backup
functions. Sometimes it is better to focus on a single aspect (or few). The
new features landing in QEMU present an opportunity. A project focused
solely on that opportunity, to work through initial set of issues, makes a
lot of sense to me. (Something like how KVM forked QEMU for a time, to
build faster x86 emulation.)

I do not see these as competing projects, and more as cooperative. The Ekko
work should be able to plug into Freezer, cleanly.

Aspects of the problem, as I see:

   1. Extracting efficient instance backups from OpenStack (via new QEMU
   function).
   2. Storing backups in efficient form (general implementation, and
   vendor-specific supersets).
   3. Offering an OpenStack backup-service API, with core and
   vendor-specific extensions.


Vendors (like my employer, EMC) might be somewhat opinionated about (2),
and for reason. :)

The huge missing piece is (1), and a focused project seems to make a lot of
sense.

As to (3), that looks like a good topic for further discussion. :)


My $.02.




On Sat, Jan 30, 2016 at 5:36 PM, Fausto Marzi 
wrote:

> Hi Preston,
> No need to apologize. They are aspect of the same problem.
> However, VMs backup is one of the many aspects that we are approaching
> here, such as:
>
> - VM backups
> - Volumes backups
> - Specific applications consistent data backup (i.e. MySQL, Mongo, file
> system, etc)
> - Provide capabilities to restore data even if keystone and swift are not
> available
> - Upload data during backup to multiple media storage in parallel
> - Web Interface
> - Provide capability to synchronize backups for sharded data on multiple
> nodes
> - Encryption
> - File based incremental
> - Block based incremental
> - Tenant related data backup and restore
> - Multi platform OS support (i.e. Linux, BSD, OSX, Windows, iOS, Android,
> etc)
> - Everything is upstreamed.
>
> This looks like a list of features... and actually it is.
>
> Block based and some multi platform OS aside, all the mentioned features
> are provided to the date. Most of them are available since Kilo.
>
> I agree with the multi API, room for vendors and to provide different
> approaches, but please let me say something (*not referring specifically
> to you or Sam or anyone*)
>
> All the time people say you have to do this and that, but the facts are
> that at the end of the day, always the same 6 engineers (not even full
> time) are working on it since 2 years, investing professional and personal
> time on it.
>
> We try to be open, to accept everybody (even the most arrogant), to
> implement features for whoever needs it, but the facts are that the only
> Companies that invested on it are HP, a bit Ericsson and Orange (apologize
> if I forgot anyone). We never said no to anyone about anything, never
> focused only to a single Company influence, never blocked a thing... and
> never will.
>
> Wouldn't be better to join efforts if companies need a backup solution and
> have their own requirements implemented by a common public Team, rather
> then start creating many tools to solve the same set of problems? How can
> ever competition benefit this? How can ever fragmenting projects help to
> provide a better solution?
>
> I'm sorry, but unless the TC or many people from the Community, tell us to
> do something different (in that case we'll do it straight away), we'll keep
> doing what we are doing, focusing on delivering what we think is the most
> advanced solution, according the resources and time we have.
>
> We need to understand that here the most important thing is to work in
> Team, to provide great tools to the Community, rather then thinking to be
> PTL or maintain independence just for the sake of it or focusing only on
> what's the best for a single Company. If this vision is not shared, then,
> unfortunately, good luck competing, while if the vision is shared... let's
> do together unprecedented things.
>
> Many thanks,
> Fausto
>
>
> On Sun, Jan 31, 2016 at 1:01 AM, Preston L. Bannister <
> pres...@bannister.us> wrote:
>
>> Seems to me there are three threads here.
>>
>> The Freezer folk were given a task, and did the best possible to support
>> backup given what OpenStack allowed. To date, OpenStack is simply not very
>> good at supporting backup as a service. (Apologies to the Freezer folk if I
>> misinterpreted.)
>>
>> The patches (finally) landing in QEMU in support of incremental backup
>> could be the basis for efficient backup services in OpenStack. This is all
>> fairly high risk, in the short term. The bits that landed in QEMU 2.4 may
>> not be sufficient (there are more QEMU patches trying to land). When put
>> into production, we may find faults. For use in OpenStack, we may 

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Fausto Marzi]

Hi Preston,
Thank you. You saw Fabrizio in Vancouver, I'm Fausto, but it's allright, : P

The challenge is interesting. If we want to build a dedicated backup API
service (which is always what we wanted to do), probably we need to:


   - Place out of Nova and Cinder the backup features, as it wouldn't make
   much sense to me to have a Backup service and also have backups managed
   independently by Nova and Cinder.


That said, I'm not a big fan of the following:

   - Interacting with the hypervisors and the volumes directly without
   passing through the Nova and Cinder API.
   - Adding any additional workload on the compute nodes or block storage
   nodes.
   - Computing incremental, compression, encryption is expensive. Have many
   simultaneous process doing that may lead  to bad behaviours on core
   services.


Some open questions:

   - Are we going to implement in Nova and Cinder, the features we need
   from the hypervisor for the backups i.e.:
  - http://wiki.qemu.org/Features/Snapshots2
  - http://wiki.qemu.org/Features/Livebackup


   - Are we going to talk directly with the hypervisors without passing
   through the services API?
   - Are we going to provide a backup service using the feature from Nova
   and Cinder API currently implemented, and improve it over time as long as
   more related advanced features are available?
   - No other service touch directly volumes and hypervisors for reasons.
   We would need to make a diamond case to get an exception for backups.
   - There's any requirement from vendors and any of them will allocate
   resources to make a backup service, as a Team?


My (flexible) thoughts are:

   - The feature is needed and is brilliant.
   - We should probably implement the newest feature provided by the
   hypervisor in Nova and export them from the Nova API.
   - Create a plugin that is integrated with Freezer to leverage that new
   features.
   - Same apply for Cinder.
   - The VMs and Volumes backup feature is already available by Nova,
   Cinder and Freezer. It needs to be improved for sure a lot, but do we need
   to create a new project for a feature that needs to be improved, rather
   than work with the existing Teams?
   - No one wants to block others, Sam proposed solution is indeed
   remarkable, but this is OpenStack, we work in Teams, why we cannot do that
   and be less fragmented.


Thanks,
Fausto


On Mon, Feb 1, 2016 at 9:23 PM, Preston L. Bannister 
wrote:

> Hi Fausto,
>
> To be clear, I am not in any way critical of Freezer and the folk putting
> in work. (Please, I want to be *entirely* clear on this point! Also, saw
> your presentation in Vancouver.)
>
> That said, Freezer is a bit of a Swiss-Army-Knife set of combined backup
> functions. Sometimes it is better to focus on a single aspect (or few). The
> new features landing in QEMU present an opportunity. A project focused
> solely on that opportunity, to work through initial set of issues, makes a
> lot of sense to me. (Something like how KVM forked QEMU for a time, to
> build faster x86 emulation.)
>
> I do not see these as competing projects, and more as cooperative. The
> Ekko work should be able to plug into Freezer, cleanly.
>
> Aspects of the problem, as I see:
>
>1. Extracting efficient instance backups from OpenStack (via new QEMU
>function).
>2. Storing backups in efficient form (general implementation, and
>vendor-specific supersets).
>3. Offering an OpenStack backup-service API, with core and
>vendor-specific extensions.
>
>
> Vendors (like my employer, EMC) might be somewhat opinionated about (2),
> and for reason. :)
>
> The huge missing piece is (1), and a focused project seems to make a lot
> of sense.
>
> As to (3), that looks like a good topic for further discussion. :)
>
>
> My $.02.
>
>
>
>
> On Sat, Jan 30, 2016 at 5:36 PM, Fausto Marzi 
> wrote:
>
>> Hi Preston,
>> No need to apologize. They are aspect of the same problem.
>> However, VMs backup is one of the many aspects that we are approaching
>> here, such as:
>>
>> - VM backups
>> - Volumes backups
>> - Specific applications consistent data backup (i.e. MySQL, Mongo, file
>> system, etc)
>> - Provide capabilities to restore data even if keystone and swift are not
>> available
>> - Upload data during backup to multiple media storage in parallel
>> - Web Interface
>> - Provide capability to synchronize backups for sharded data on multiple
>> nodes
>> - Encryption
>> - File based incremental
>> - Block based incremental
>> - Tenant related data backup and restore
>> - Multi platform OS support (i.e. Linux, BSD, OSX, Windows, iOS, Android,
>> etc)
>> - Everything is upstreamed.
>>
>> This looks like a list of features... and actually it is.
>>
>> Block based and some multi platform OS aside, all the mentioned features
>> are provided to the date. Most of them are available since Kilo.
>>
>> I agree with the multi API, room for vendors and 

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Mon, Feb 1, 2016 at 8:23 PM, Preston L. Bannister 
wrote:

> Hi Fausto,
>
> To be clear, I am not in any way critical of Freezer and the folk putting
> in work. (Please, I want to be *entirely* clear on this point! Also, saw
> your presentation in Vancouver.)
>
> That said, Freezer is a bit of a Swiss-Army-Knife set of combined backup
> functions. Sometimes it is better to focus on a single aspect (or few). The
> new features landing in QEMU present an opportunity. A project focused
> solely on that opportunity, to work through initial set of issues, makes a
> lot of sense to me. (Something like how KVM forked QEMU for a time, to
> build faster x86 emulation.)
>
> I do not see these as competing projects, and more as cooperative. The
> Ekko work should be able to plug into Freezer, cleanly.
>

>From what I see this is certainly a possibly future. Ekko may be able to
complement Freezer by running as a plugin, but just as easily be a
standalone project that is fully compatible with a Freezer without
conflicting in any way. As an operator, I am a fan of only deploying what
is needed and Freezer needs infrastructure to do what it does that isn't
useful to block-based backup purposed by Ekko. That may change as we have
already started talking with the Freezer team and they have this idea of a
plugin-type system that may very well do the trick.


>
Aspects of the problem, as I see:
>
>1. Extracting efficient instance backups from OpenStack (via new QEMU
>function).
>2. Storing backups in efficient form (general implementation, and
>vendor-specific supersets).
>3. Offering an OpenStack backup-service API, with core and
>vendor-specific extensions.
>
>
> Vendors (like my employer, EMC) might be somewhat opinionated about (2),
> and for reason. :)
>

On the note of point (2), its not so much the storage as managing retention
in object storage (essentially a read-only medium). I would argue (2) is
much harder than (1). People have been doing (1) with VMWare for a long
time, and QEMU's steps won't be that different.


> The huge missing piece is (1), and a focused project seems to make a lot
> of sense.
>

And everyone please keep in mind, the only overlap I have seen so far is
the API, and _potentially_ the scheduler. So if a merging is needed, then
it should be fairly simple. None of this has to be decided right now. We
aren't going down a path that can't be reversed, and we likely never will
given how little these two projects overlap in their current forms.

Sam Yaple
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Mon, Feb 1, 2016 at 10:32 PM, Fausto Marzi 
wrote:

> Hi Preston,
> Thank you. You saw Fabrizio in Vancouver, I'm Fausto, but it's allright, :
> P
>
> The challenge is interesting. If we want to build a dedicated backup API
> service (which is always what we wanted to do), probably we need to:
>
>
>- Place out of Nova and Cinder the backup features, as it wouldn't
>make much sense to me to have a Backup service and also have backups
>managed independently by Nova and Cinder.
>
>
> That said, I'm not a big fan of the following:
>
>- Interacting with the hypervisors and the volumes directly without
>passing through the Nova and Cinder API.
>
> Passing through the api will be a huge issue for extracting data due to
the sheer volume of data needed (TB through the api is going to kill
everything!)

>
>- Adding any additional workload on the compute nodes or block storage
>nodes.
>- Computing incremental, compression, encryption is expensive. Have
>many simultaneous process doing that may lead  to bad behaviours on core
>services.
>
> These are valid concerns, but the alternative is still shipping the raw
data elsewhere to do this work, and that has its own issue in terms of
bandwidth.

>
> My (flexible) thoughts are:
>
>- The feature is needed and is brilliant.
>- We should probably implement the newest feature provided by the
>hypervisor in Nova and export them from the Nova API.
>- Create a plugin that is integrated with Freezer to leverage that new
>features.
>- Same apply for Cinder.
>- The VMs and Volumes backup feature is already available by Nova,
>Cinder and Freezer. It needs to be improved for sure a lot, but do we need
>to create a new project for a feature that needs to be improved, rather
>than work with the existing Teams?
>
> I disagree with this statement strongly as I have stated before. Nova has
snapshots. Cinder has snapshots (though they do say cinder-backup). Freezer
wraps Nova and Cinder. Snapshots are not backups. They are certainly not
_incremental_ backups. They can have neither compression, nor encryption.
With this in mind, Freezer does not have this "feature" at all. Its not
that it needs improvement, it simply does not exist in Freezer. So a
separate project dedicated to that one goal is not unreasonable. The real
question is whether it is practical to merge Freezer and Ekko, and this is
the question Ekko and the Freezer team are attempting to answer.

>
>- No one wants to block others, Sam proposed solution is indeed
>remarkable, but this is OpenStack, we work in Teams, why we cannot do that
>and be less fragmented.
>
>
> Thanks,
> Fausto
>
>
Sam Yaple
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Thierry Carrez]

Fausto Marzi wrote:

[...]
That said, what we are trying to do, with Sam (that is showing an open
attitude and constructive approach), is to understand if the solution
make sense itself, if we can work together as a Team and if it make
sense to include it in Freezer. If not, it is crystal clear, that no one
in the world, can stop a very committed and capable person to do what
he/she wants to do. Even less in the Open Source and here. We probably
need to avoid fragmentation and at the same time no limiting new ideas,
but potentiate them.


It's also worth noting that Freezer is currently an official OpenStack 
project while Ekko is not, so there is no real overlap (yet).


If Ekko wants one day to become an official project, it would be great 
to clarify its positioning relative to Freezer before it applies, as I 
expect that will be the first question from the Technical Committee.


--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Fausto Marzi]

Hi Preston,
No need to apologize. They are aspect of the same problem.
However, VMs backup is one of the many aspects that we are approaching
here, such as:

- VM backups
- Volumes backups
- Specific applications consistent data backup (i.e. MySQL, Mongo, file
system, etc)
- Provide capabilities to restore data even if keystone and swift are not
available
- Upload data during backup to multiple media storage in parallel
- Web Interface
- Provide capability to synchronize backups for sharded data on multiple
nodes
- Encryption
- File based incremental
- Block based incremental
- Tenant related data backup and restore
- Multi platform OS support (i.e. Linux, BSD, OSX, Windows, iOS, Android,
etc)
- Everything is upstreamed.

This looks like a list of features... and actually it is.

Block based and some multi platform OS aside, all the mentioned features
are provided to the date. Most of them are available since Kilo.

I agree with the multi API, room for vendors and to provide different
approaches, but please let me say something (*not referring specifically to
you or Sam or anyone*)

All the time people say you have to do this and that, but the facts are
that at the end of the day, always the same 6 engineers (not even full
time) are working on it since 2 years, investing professional and personal
time on it.

We try to be open, to accept everybody (even the most arrogant), to
implement features for whoever needs it, but the facts are that the only
Companies that invested on it are HP, a bit Ericsson and Orange (apologize
if I forgot anyone). We never said no to anyone about anything, never
focused only to a single Company influence, never blocked a thing... and
never will.

Wouldn't be better to join efforts if companies need a backup solution and
have their own requirements implemented by a common public Team, rather
then start creating many tools to solve the same set of problems? How can
ever competition benefit this? How can ever fragmenting projects help to
provide a better solution?

I'm sorry, but unless the TC or many people from the Community, tell us to
do something different (in that case we'll do it straight away), we'll keep
doing what we are doing, focusing on delivering what we think is the most
advanced solution, according the resources and time we have.

We need to understand that here the most important thing is to work in
Team, to provide great tools to the Community, rather then thinking to be
PTL or maintain independence just for the sake of it or focusing only on
what's the best for a single Company. If this vision is not shared, then,
unfortunately, good luck competing, while if the vision is shared... let's
do together unprecedented things.

Many thanks,
Fausto


On Sun, Jan 31, 2016 at 1:01 AM, Preston L. Bannister 
wrote:

> Seems to me there are three threads here.
>
> The Freezer folk were given a task, and did the best possible to support
> backup given what OpenStack allowed. To date, OpenStack is simply not very
> good at supporting backup as a service. (Apologies to the Freezer folk if I
> misinterpreted.)
>
> The patches (finally) landing in QEMU in support of incremental backup
> could be the basis for efficient backup services in OpenStack. This is all
> fairly high risk, in the short term. The bits that landed in QEMU 2.4 may
> not be sufficient (there are more QEMU patches trying to land). When put
> into production, we may find faults. For use in OpenStack, we may need
> changes in libvirt, and/or in Nova. (Or *maybe* not if usage for backup
> proves orthogonal.)  The only way to work out the prior is to start. The
> timeline could be months or years.
>
> There is a need for a common API for backup as a service in the cloud.
> Something more than imitating AWS. Allow some room for vendors with
> differing approach.
>
> I see the above as not competing, but aspects of the same problem.
>
>
> ​
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Preston L. Bannister]

Seems to me there are three threads here.

The Freezer folk were given a task, and did the best possible to support
backup given what OpenStack allowed. To date, OpenStack is simply not very
good at supporting backup as a service. (Apologies to the Freezer folk if I
misinterpreted.)

The patches (finally) landing in QEMU in support of incremental backup
could be the basis for efficient backup services in OpenStack. This is all
fairly high risk, in the short term. The bits that landed in QEMU 2.4 may
not be sufficient (there are more QEMU patches trying to land). When put
into production, we may find faults. For use in OpenStack, we may need
changes in libvirt, and/or in Nova. (Or *maybe* not if usage for backup
proves orthogonal.)  The only way to work out the prior is to start. The
timeline could be months or years.

There is a need for a common API for backup as a service in the cloud.
Something more than imitating AWS. Allow some room for vendors with
differing approach.

I see the above as not competing, but aspects of the same problem.


​
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Fausto Marzi]

What motivates me every day and every night, is to provide the most
advanced solution for a set a problems to solve, Open Source and in
OpenStack. What motivates me is to work with brilliant people like minded,
capable of doing great things working together. It is not the competition
and It is not the PTL or Core label that gives you that.

In OpenStack most of the time, new project born because there's a group of
people in a company that needs to provide a solution for a set of problems,
and the same people/company allocate resources, time and efforts. Now if
every company start his own project, we are likely to going to have many
individual tools to provide a similar set of solutions. Is this what we
want? I hope not.

Starting new projects is a tremendously hard work and with limited
resources most of the time. So why not join, work as a community, provide a
remarkable service/tool and secure the result for who decide to use
OpenStack.

We have to do the effort of trying hard to work together, smooth
differences, improve others rather than block them. Honestly as a PTL I try
to do that, every single bloody day, dealing with managers, companies,
customers, engineers, code, bugs and community. However I do not think a
PTL should be elected 2 consecutive times, for official projects, because
it's a great experience that anyone that deserve it, should learn from. But
this is just my opinion.

That said, what we are trying to do, with Sam (that is showing an open
attitude and constructive approach), is to understand if the solution make
sense itself, if we can work together as a Team and if it make sense to
include it in Freezer. If not, it is crystal clear, that no one in the
world, can stop a very committed and capable person to do what he/she wants
to do. Even less in the Open Source and here. We probably need to avoid
fragmentation and at the same time no limiting new ideas, but potentiate
them.

Happy to hear your thoughts on this.

Thanks,
Fausto



On Thu, Jan 28, 2016 at 8:55 PM, Ian Wells  wrote:

> On 27 January 2016 at 11:06, Flavio Percoco  wrote:
>
>> FWIW, the current governance model does not prevent competition. That's
>> not to
>> be understood as we encourage it but rather than there could be services
>> with
>> some level of overlap that are still worth being separate.
>>
>
> There should always be the possibility to compete; it's always possible
> that rethinking an idea produces a better implementation of the same API.
> But we don't separate API from implementation in Openstack - the 'XXX API'
> cannot easily be divorced from the project containing the implementation
> when the definition of the 'XXX API' is 'the API implemented by the XXX
> code'.  We should separate them - the API is the only thing that a tenant
> will ever actually care about, not the implementation choice behind it.
>
> What Jay is referring to is that regardless the projects do similar
>> things, the
>> same or totally different things, we should strive to have different
>> APIs. The
>> API shouldn't overlap in terms of endpoints and the way they are exposed.
>>
>
> And for this, perhaps we should have an API namespace registry, so that
> when two groups implement the same endpoints they have to implement the
> same API?  I think Jay's point is that we muddy the waters here by having
> confusingly similar-but-different APIs.
>
> [The counterargument is that service discovery usually determines what API
> you're getting, so that if two services claim to be different service types
> in Keystone then they are *not* the same API and should be allowed free
> reign of their URI namespace, but I see that's not working for us.]
>
> And, coming back to the original point, if Freezer and Ekko both implement
> backups, and they can come to an agreement on what 'a backup' is and an API
> definition for it, that means that they could exist as independent projects
> with independent codebases that both implement /backup - but, importantly,
> in a consistent way that doesn't confuse app developers.  That will only
> work if the API definition stands separate from the projects, though.
>
> --
> Ian.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Ian Wells]

On 27 January 2016 at 11:06, Flavio Percoco  wrote:

> FWIW, the current governance model does not prevent competition. That's
> not to
> be understood as we encourage it but rather than there could be services
> with
> some level of overlap that are still worth being separate.
>

There should always be the possibility to compete; it's always possible
that rethinking an idea produces a better implementation of the same API.
But we don't separate API from implementation in Openstack - the 'XXX API'
cannot easily be divorced from the project containing the implementation
when the definition of the 'XXX API' is 'the API implemented by the XXX
code'.  We should separate them - the API is the only thing that a tenant
will ever actually care about, not the implementation choice behind it.

What Jay is referring to is that regardless the projects do similar things,
> the
> same or totally different things, we should strive to have different APIs.
> The
> API shouldn't overlap in terms of endpoints and the way they are exposed.
>

And for this, perhaps we should have an API namespace registry, so that
when two groups implement the same endpoints they have to implement the
same API?  I think Jay's point is that we muddy the waters here by having
confusingly similar-but-different APIs.

[The counterargument is that service discovery usually determines what API
you're getting, so that if two services claim to be different service types
in Keystone then they are *not* the same API and should be allowed free
reign of their URI namespace, but I see that's not working for us.]

And, coming back to the original point, if Freezer and Ekko both implement
backups, and they can come to an agreement on what 'a backup' is and an API
definition for it, that means that they could exist as independent projects
with independent codebases that both implement /backup - but, importantly,
in a consistent way that doesn't confuse app developers.  That will only
work if the API definition stands separate from the projects, though.

-- 
Ian.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Jay Pipes]

On 01/28/2016 06:03 AM, Steven Dake (stdake) wrote:

The TC set precedent in this first competition-induced review which is
worth a good read for other projects thinking of competing with existing
projects of which there are already plenty in OpenStack..

https://review.openstack.org/206789


Steven, my point in this thread is not about competition within a 
problem domain. It was specifically about overlapping REST API resource 
endpoints.


Kolla does not have a REST API.

Neither does Triple-O. Or OSAD. Or Puppet OpenStack. Or the OpenStack 
Chef Cookbooks.


So, I am failing to see that the above governance patch is indeed a 
precedent for this conversation. Rather, it was the governance patch 
that added Monasca as an official OpenStack project that I brought up 
the problem of the Monasca's API overlap with the OpenStack Telemetry 
API (which was Ceilometer's API at the time):


https://review.openstack.org/#/c/213183/

It was Joe Gordon and others who advised that there was nothing in the 
project acceptance criteria that mentioned a lack of API overlap as a 
condition for inclusion.


I think there should definitely be some requirement added to the project 
acceptance criteria, but I'm in the minority it seems in this thinking.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[gordon chung]

On 28/01/2016 1:03 AM, Steven Dake (stdake) wrote:
> Choice is one of life's fundamental elements that make life interesting.
> Let the Operators choose what they wish based upon technical merits rather
> then "who was first to publish a project".

yes, it's nice when competing entities exists in the same domain... i'm 
not sure it's the same when it's also within the same community. i 
realise this is common practice in big enterprise where often they'll 
make multiple plays on the same thing because a) they built up a lot 
money in the past and b) they are too big and didn't realise someone 
else was doing it as well. regardless, it's quite obvious having 
multiple plays does split up your best talent and it's debatable whether 
the surviving solution is the best solution or just a solution. it also 
devalues the work each team is doing so workers care less about it but 
that's a psychological argument which i don't have a survey to reference.

regardless if it's creative and original, my question would be why was 
your first step to create something new rather than advocate for change?

i'll go a slight tangent because i feel like this is one of the root of 
the issues: i think we need to devalue what a PTL is. there are many 
companies who seek PTLs and even promote the number of PTLs on staff as 
if they are some infallible deity. this is not what a PTL is and you are 
doing it wrong if that's what you think it is. can we rebrand PTL to 
what it is? team administrator or secretary or liaison (wow, this is 
actually useful now)? maybe this will stop people from trying so hard to 
have a PTL in their title and they will attempt to collaborate first 
rather than duplicate.

On 28/01/2016 6:34 AM, Chris Dent wrote:
> Let's be honest: From my perspective Joe (and others) had it in for
> Ceilometer and was hoping that the validation of Monasca would have some
> impact on improving the Telemetry situation, so the lack of criteria
> that would have prevent Monasca's inclusion is convenient. 

go back to silly-ometer dude :P

cheers,

-- 
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Wed, Jan 27, 2016 at 8:19 PM, Fausto Marzi 
wrote:

> Hi Sam,
>
> After our conversation, I have few questions and consideration about Ekko,
> mainly on how it works et similar. Also to make available to the community
> our discussions:
>
> -  In understand you are placing a backup-agent on the compute
> node and execute actions interacting directly with the hypervisor. I’m
> thinking that while Ekko execute this actions, the Nova service have no
> visibility whatsoever of this. I do not think is a good idea to execute
> actions directly on the hypervisor without interacting with the Nova API.
>
This is not an ideal situation, no. Nova should be aware of what we are
doing and when we are doing it. We are aware of this and plan on purposing
ways to be better integrated with Nova (and Cinder for that matter).

> -  On your assumptions, you said that Nova snapshots creation
> generate a VM downtime. I don’t think the assumption is correct, at least
> in Kilo, Liberty and Mitaka. The only downtime you may have related to the
> snapshot, is when you merge back the snapshot to the original root image,
> and this is not our case here.
>
For Kilo and up Nova does leverage the live snapshot allowing for a
snapshot without placing the instance into a Paused state. That is correct.
Some of the same underlying functions are used for the IncrementalBackup
feature of QEMU as well. So you are right its not fair to say that
snapshots always cause downtime as that hasn't been the case since Kilo.

> -  How the restore would work? If you do a restore of the VM and
> the record of that VM instance is not available in the Nova DB (i.e.
> restoring a VM on a newly installed Openstack cloud, or in another
> region, or after a vm has beed destroyed)what would happen? How do you
> manage the consistency of the data between Nova DB and VM status
>
Restore has two pieces. The one we are definitely implementing is restoring
an backup image to a glance image. At that point anyone could start an
instance off of it. Additionally, it _could_ be restored directly back to
the instance in question by powering off the instance and restoring the
data directly back then starting the instance again.

> -  If you execute a backup of the VM image file without executing
> a backup of the related VM metadata information (in the shortest time frame
> as possible) there are chances the backup can be inconsistent.
>
I don't see how VM metadata information has anything to do with a proper
backup of the data in this case.

> - How the restore would happen if on that moment Keystone or Swift is not
> available?
>
How does anything happen if Keystone isn't available? User can't auth so
nothing happens.

> -  Does the backup that Ekko execute, generates bootable image?
> If not, the image is not usable and the restore process will take longer to
> execute the steps to make the image bootable.
>
The backup Ekko would take would be a bit-for-bit copy of what is on the
underlying disk. If that is bootable, the it is bootable.

> -   I do not see any advantage in Ekko over using Nova API to
> snapshot -> Generate an image -> upload to Glance -> upload to Swift.
>
Snapshots are not backups. This is a very important point. Additionally the
process you describe is extremely expensive in terms of time, bandwidth,
and IO. For sake of example, if you have 1TB of data on an instance and you
snapshot it you must upload 1TB to Glance/Swift. The following day you do
another snapshot and you must upload another 1TB, likely the majority of
the data is exactly the same. With Ekko (or any true backup) you should
only be uploading what has changed since the last backup.

> -  The Ekko approach is limited to Nova, KVM QEMU, having a
> qemu-agent running on the VM. I think the scope is probably a bit limited.
> This is more a feature than a tool itself, but the problem is being solved
> I think more efficiently already.
>
It is not limited to Nova, Libvirt nor QEMU. It also does not _require_ the
qemu-agent. It can be used independent of OpenStack (though that is not the
endgoal) and even with VMWare or Hyper-V since they both support CBT which
is the main component we leverage.

> -  By executing all the actions related to backup (i.e.
> compression, incremental computation, upload, I/O and segmented upload to
> Swift) Ekko is adding a significant load to the Compute Nodes. All the
> work is done on the hypervisor and not taken into account by ceilometer (or
> similar), so for example not billable. I do not think this is a good idea
> as distributing the load over multiple components helps OpenStack to scale
> and by leveraging the existing API you integrated better with existing
> tools.
>
The backup-agent that is purposed to exist on the compute node does not
necessarily perform the upload from the compute node since the data may
exist on a backend like Ceph or NFS. But in the case 

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Steven Dake (stdake)]

On 1/27/16, 12:06 PM, "Flavio Percoco"  wrote:

>On 27/01/16 12:16 -0500, Emilien Macchi wrote:
>>
>>
>>On 01/27/2016 10:51 AM, Jay Pipes wrote:
>>> On 01/27/2016 12:53 PM, gordon chung wrote:
> It makes for a crappy user experience. Crappier than the crappy user
> experience that OpenStack API users already have because we have
>done a
> crappy job shepherding projects in order to make sure there isn't
> overlap between their APIs (yes, Ceilometer and Monasca, I'm looking
> directly at you).
 ... yes, Ceilometer can easily handle your events and meters and store
 them in either Elasticsearch or Gnocchi for visualisations. you just
 need to create a new definition in our mapping files[1][2]. you will
 definitely want to coordinate the naming of your messages. ie.
 event_type == backup. and event_type ==
 backup..
>>>
>>> This isn't at all what I was referring to, actually. I was referring to
>>> my belief that we (the API WG, the TC, whatever...) have failed to
>>> properly prevent almost complete and total overlap of the Ceilometer
>>>[1]
>>> and Monasca [2] REST APIs.
>>>
>>> They are virtually identical in purpose, but in frustrating
>>> slightly-inconsistent ways. and this means that users of the "OpenStack
>>> APIs" have absolutely no idea what the "OpenStack Telemetry API"
>>>really is.
>>>
>>> Both APIs have /alarms as a top-level resource endpoint. One of them
>>> refers to the alarm notification with /alarms, while the other refers
>>>to
>>> the alarm definition with /alarms.
>>>
>>> One API has /meters as a top-level resource endpoint. The other uses
>>> /metrics to mean the exact same thing.
>>>
>>> One API has /samples as a top-level resource endpoint. The other uses
>>> /metrics/measurements to mean the exact same thing.
>>>
>>> One API returns a list JSON object for list results. The other returns
>>>a
>>> dict JSON object with a "links" key and an "elements" key.
>>>
>>> And the list goes on... all producing a horrible non-unified,
>>> overly-complicated and redundant experience for our API users.
>>>
>>
>>I agree with you here Jay, Monasca is a great example of failure in
>>having consistency across OpenStack projects.
>>It's a different topic but maybe a retrospective of what happened could
>>help our community to not reproduce the same mistakes again.
>>
>>Please do not repeat this failure for other projects.
>>Do not duplicate efforts: if Ekko has a similar mission statement, maybe
>>we should avoid creating a new project and contribute to Freezer?
>>(I'm probably missing some technical bits so tell me if I'm wrong)
>
>FWIW, the current governance model does not prevent competition. That's
>not to
>be understood as we encourage it but rather than there could be services
>with
>some level of overlap that are still worth being separate.
>
>What Jay is referring to is that regardless the projects do similar
>things, the
>same or totally different things, we should strive to have different
>APIs. The
>API shouldn't overlap in terms of endpoints and the way they are exposed.
>
>With all that said, I'd like to encourage collaboration over competition
>and I'm
>sure both teams will find a way to make this work.
>
>Cheers,
>Flavio
>
>
>-- 
>@flaper87
>Flavio Percoco

Flavio,

Of course you know this, but for the broader community that may not be
aware, the exact governance repo line item is as follows:
"

* Where it makes sense, the project cooperates with existing projects
rather than gratuitously competing or reinventing the wheel

"

Now that line could be interpreted in many ways, but when Kolla went
through incubation with at least 5 other competitors in the deployment
space, the issue of competition came up and the TC argued that competition
was a good thing on the outer-layer services (such as deployment) and a
bad thing for the inner layer services (such as nova).  The fact that APIs
may be duplicated in some way is not an optimal situation, but if that is
what the TC wishes, the governance repository for new projects should be
updated to indicate the guidelines.

Sam and the EKKO core team's work is creative, original, and completely
different then freezer.  The only thing they have in common is they both
do backups.  They fundamentally operate in different ways.

The TC set precedent in this first competition-induced review which is
worth a good read for other projects thinking of competing with existing
projects of which there are already plenty in OpenStack..

https://review.openstack.org/206789


My parsing of the Technical Committee precedent set there is if the
project is technically different in significant ways, then its A-OK for
big-tent inclusion down the road and definitely suitable for a new project
development effort.

Sam and the EKKO core team's work is creative, original, and completely
different then freezer.  The only thing they have in common is they both
do backups.  They fundamentally operate in 

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Jay Pipes]

On 01/27/2016 12:53 PM, gordon chung wrote:

It makes for a crappy user experience. Crappier than the crappy user
experience that OpenStack API users already have because we have done a
crappy job shepherding projects in order to make sure there isn't
overlap between their APIs (yes, Ceilometer and Monasca, I'm looking
directly at you).

... yes, Ceilometer can easily handle your events and meters and store
them in either Elasticsearch or Gnocchi for visualisations. you just
need to create a new definition in our mapping files[1][2]. you will
definitely want to coordinate the naming of your messages. ie.
event_type == backup. and event_type == backup..


This isn't at all what I was referring to, actually. I was referring to 
my belief that we (the API WG, the TC, whatever...) have failed to 
properly prevent almost complete and total overlap of the Ceilometer [1] 
and Monasca [2] REST APIs.


They are virtually identical in purpose, but in frustrating 
slightly-inconsistent ways. and this means that users of the "OpenStack 
APIs" have absolutely no idea what the "OpenStack Telemetry API" really is.


Both APIs have /alarms as a top-level resource endpoint. One of them 
refers to the alarm notification with /alarms, while the other refers to 
the alarm definition with /alarms.


One API has /meters as a top-level resource endpoint. The other uses 
/metrics to mean the exact same thing.


One API has /samples as a top-level resource endpoint. The other uses 
/metrics/measurements to mean the exact same thing.


One API returns a list JSON object for list results. The other returns a 
dict JSON object with a "links" key and an "elements" key.


And the list goes on... all producing a horrible non-unified, 
overly-complicated and redundant experience for our API users.


Best,
-jay

[1] http://developer.openstack.org/api-ref-telemetry-v2.html
[2] 
https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Emilien Macchi]

On 01/27/2016 10:51 AM, Jay Pipes wrote:
> On 01/27/2016 12:53 PM, gordon chung wrote:
>>> It makes for a crappy user experience. Crappier than the crappy user
>>> experience that OpenStack API users already have because we have done a
>>> crappy job shepherding projects in order to make sure there isn't
>>> overlap between their APIs (yes, Ceilometer and Monasca, I'm looking
>>> directly at you).
>> ... yes, Ceilometer can easily handle your events and meters and store
>> them in either Elasticsearch or Gnocchi for visualisations. you just
>> need to create a new definition in our mapping files[1][2]. you will
>> definitely want to coordinate the naming of your messages. ie.
>> event_type == backup. and event_type ==
>> backup..
> 
> This isn't at all what I was referring to, actually. I was referring to
> my belief that we (the API WG, the TC, whatever...) have failed to
> properly prevent almost complete and total overlap of the Ceilometer [1]
> and Monasca [2] REST APIs.
> 
> They are virtually identical in purpose, but in frustrating
> slightly-inconsistent ways. and this means that users of the "OpenStack
> APIs" have absolutely no idea what the "OpenStack Telemetry API" really is.
> 
> Both APIs have /alarms as a top-level resource endpoint. One of them
> refers to the alarm notification with /alarms, while the other refers to
> the alarm definition with /alarms.
> 
> One API has /meters as a top-level resource endpoint. The other uses
> /metrics to mean the exact same thing.
> 
> One API has /samples as a top-level resource endpoint. The other uses
> /metrics/measurements to mean the exact same thing.
> 
> One API returns a list JSON object for list results. The other returns a
> dict JSON object with a "links" key and an "elements" key.
> 
> And the list goes on... all producing a horrible non-unified,
> overly-complicated and redundant experience for our API users.
> 

I agree with you here Jay, Monasca is a great example of failure in
having consistency across OpenStack projects.
It's a different topic but maybe a retrospective of what happened could
help our community to not reproduce the same mistakes again.

Please do not repeat this failure for other projects.
Do not duplicate efforts: if Ekko has a similar mission statement, maybe
we should avoid creating a new project and contribute to Freezer?
(I'm probably missing some technical bits so tell me if I'm wrong)

As an operator, I don't want see 2 OpenStack projects solving the same
issue.
As a developer, I don't want to implement the same feature in 2
different projects.

If we have (again) 2 projects with the same mission statement, I think
we'll waste time & resources, and eventually isolate people working on
their own projects.

I'm sure we don't want that.
-- 
Emilien Macchi



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[gordon chung]

On 27/01/2016 10:51 AM, Jay Pipes wrote:
> On 01/27/2016 12:53 PM, gordon chung wrote:
>>> It makes for a crappy user experience. Crappier than the crappy user
>>> experience that OpenStack API users already have because we have done a
>>> crappy job shepherding projects in order to make sure there isn't
>>> overlap between their APIs (yes, Ceilometer and Monasca, I'm looking
>>> directly at you).
>> ... yes, Ceilometer can easily handle your events and meters and store
>> them in either Elasticsearch or Gnocchi for visualisations. you just
>> need to create a new definition in our mapping files[1][2]. you will
>> definitely want to coordinate the naming of your messages. ie.
>> event_type == backup. and event_type == 
>> backup..
>
> This isn't at all what I was referring to, actually. I was referring 
> to my belief that we (the API WG, the TC, whatever...) have failed to 
> properly prevent almost complete and total overlap of the Ceilometer 
> [1] and Monasca [2] REST APIs.
>
> They are virtually identical in purpose, but in frustrating 
> slightly-inconsistent ways. and this means that users of the 
> "OpenStack APIs" have absolutely no idea what the "OpenStack Telemetry 
> API" really is.
>
> Both APIs have /alarms as a top-level resource endpoint. One of them 
> refers to the alarm notification with /alarms, while the other refers 
> to the alarm definition with /alarms.
>
> One API has /meters as a top-level resource endpoint. The other uses 
> /metrics to mean the exact same thing.
>
> One API has /samples as a top-level resource endpoint. The other uses 
> /metrics/measurements to mean the exact same thing.
>
> One API returns a list JSON object for list results. The other returns 
> a dict JSON object with a "links" key and an "elements" key.
>
> And the list goes on... all producing a horrible non-unified, 
> overly-complicated and redundant experience for our API users.
>
> Best,
> -jay
>
> [1] http://developer.openstack.org/api-ref-telemetry-v2.html
> [2] 
> https://github.com/openstack/monasca-api/blob/master/docs/monasca-api-spec.md
>
> __ 
>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

... i'm aware, thus the leading dots. as i saw no suggestions in your 
message -- just a statement -- i chose to provide some 'hopefully' 
constructive comments rather than make assumption on what you were 
hinting at. obviously, no one is able to foresee the existence of a 
project built internally within another company let alone the api of 
said project, i'm not sure what the proposed resolution is.

as the scope is different between ekko and freezer (same for monasca and 
telemetry according to governance voting[1]) what is the issue with 
having overlaps? bringing in CADF[2], if you define your taxonomy 
correctly, sharing a common base is fine as long as there exists enough 
granularity in how you define your resources that differentiates a 
'freezer' resource and an 'ekko' resource. that said, if they are truly 
the same, then you should probably be debating why you have two of the 
same thing instead of api.

[1] https://review.openstack.org/#/c/213183/
[2] 
https://www.dmtf.org/sites/default/files/standards/documents/DSP0262_1.0.0.pdf

-- 
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Flavio Percoco]

On 27/01/16 12:16 -0500, Emilien Macchi wrote:



On 01/27/2016 10:51 AM, Jay Pipes wrote:

On 01/27/2016 12:53 PM, gordon chung wrote:

It makes for a crappy user experience. Crappier than the crappy user
experience that OpenStack API users already have because we have done a
crappy job shepherding projects in order to make sure there isn't
overlap between their APIs (yes, Ceilometer and Monasca, I'm looking
directly at you).

... yes, Ceilometer can easily handle your events and meters and store
them in either Elasticsearch or Gnocchi for visualisations. you just
need to create a new definition in our mapping files[1][2]. you will
definitely want to coordinate the naming of your messages. ie.
event_type == backup. and event_type ==
backup..


This isn't at all what I was referring to, actually. I was referring to
my belief that we (the API WG, the TC, whatever...) have failed to
properly prevent almost complete and total overlap of the Ceilometer [1]
and Monasca [2] REST APIs.

They are virtually identical in purpose, but in frustrating
slightly-inconsistent ways. and this means that users of the "OpenStack
APIs" have absolutely no idea what the "OpenStack Telemetry API" really is.

Both APIs have /alarms as a top-level resource endpoint. One of them
refers to the alarm notification with /alarms, while the other refers to
the alarm definition with /alarms.

One API has /meters as a top-level resource endpoint. The other uses
/metrics to mean the exact same thing.

One API has /samples as a top-level resource endpoint. The other uses
/metrics/measurements to mean the exact same thing.

One API returns a list JSON object for list results. The other returns a
dict JSON object with a "links" key and an "elements" key.

And the list goes on... all producing a horrible non-unified,
overly-complicated and redundant experience for our API users.



I agree with you here Jay, Monasca is a great example of failure in
having consistency across OpenStack projects.
It's a different topic but maybe a retrospective of what happened could
help our community to not reproduce the same mistakes again.

Please do not repeat this failure for other projects.
Do not duplicate efforts: if Ekko has a similar mission statement, maybe
we should avoid creating a new project and contribute to Freezer?
(I'm probably missing some technical bits so tell me if I'm wrong)


FWIW, the current governance model does not prevent competition. That's not to
be understood as we encourage it but rather than there could be services with
some level of overlap that are still worth being separate.

What Jay is referring to is that regardless the projects do similar things, the
same or totally different things, we should strive to have different APIs. The
API shouldn't overlap in terms of endpoints and the way they are exposed.

With all that said, I'd like to encourage collaboration over competition and I'm
sure both teams will find a way to make this work.

Cheers,
Flavio


--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Wed, Jan 27, 2016 at 7:06 PM, Flavio Percoco  wrote:
>
> FWIW, the current governance model does not prevent competition. That's
> not to
> be understood as we encourage it but rather than there could be services
> with
> some level of overlap that are still worth being separate.
>
> What Jay is referring to is that regardless the projects do similar
> things, the
> same or totally different things, we should strive to have different APIs.
> The
> API shouldn't overlap in terms of endpoints and the way they are exposed.
>
> With all that said, I'd like to encourage collaboration over competition
> and I'm
> sure both teams will find a way to make this work.
>
> Cheers,
> Flavio


And to come full circle on this thread, I will point out once again there
is no competition between Ekko and Freezer at this time. Freezer is
file-level backup where Ekko is block-level backup. Anyone familiar with
backups knows these are drastically different types of backups. Those using
block-level backups typically won't be using file-level backups and
vice-versa. That said, even if there is no convergence of Freezer and Ekko
 they can still live side-by-side without any conflict at all.

As of now, Ekko and Freezer teams have started a dialogue and we will
continue to collaborate rather than compete in every way that is reasonable
for both projects.

Sam Yaple
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Fausto Marzi]

Hi Sam,

After our conversation, I have few questions and consideration about Ekko,
mainly on how it works et similar. Also to make available to the community
our discussions:

-  In understand you are placing a backup-agent on the compute node
and execute actions interacting directly with the hypervisor. I’m thinking
that while Ekko execute this actions, the Nova service have no visibility
whatsoever of this. I do not think is a good idea to execute actions
directly on the hypervisor without interacting with the Nova API.

-  On your assumptions, you said that Nova snapshots creation
generate a VM downtime. I don’t think the assumption is correct, at least
in Kilo, Liberty and Mitaka. The only downtime you may have related to the
snapshot, is when you merge back the snapshot to the original root image,
and this is not our case here.

-  How the restore would work? If you do a restore of the VM and
the record of that VM instance is not available in the Nova DB (i.e.
restoring a VM on a newly installed Openstack cloud, or in another region,
or after a vm has beed destroyed)what would happen? How do you manage the
consistency of the data between Nova DB and VM status

-  If you execute a backup of the VM image file without executing a
backup of the related VM metadata information (in the shortest time frame
as possible) there are chances the backup can be inconsistent.

- How the restore would happen if on that moment Keystone or Swift is not
available?

-  Does the backup that Ekko execute, generates bootable image? If
not, the image is not usable and the restore process will take longer to
execute the steps to make the image bootable.

-   I do not see any advantage in Ekko over using Nova API to
snapshot -> Generate an image -> upload to Glance -> upload to Swift.

-  The Ekko approach is limited to Nova, KVM QEMU, having a
qemu-agent running on the VM. I think the scope is probably a bit limited.
This is more a feature than a tool itself, but the problem is being solved
I think more efficiently already.

-  By executing all the actions related to backup (i.e.
compression, incremental computation, upload, I/O and segmented upload to
Swift) Ekko is adding a significant load to the Compute Nodes. All the work
is done on the hypervisor and not taken into account by ceilometer (or
similar), so for example not billable. I do not think this is a good idea
as distributing the load over multiple components helps OpenStack to scale
and by leveraging the existing API you integrated better with existing
tools.

-  There’s no documentation whatsoever provided with Ekko. I had to
read the source code, have conversations directly with you and invest
significant time on it. I think provide some documentation is helpful, as
the doc link in the openstack/ekko repo return 404 Not Found.

Please let me know what your thoughts are on this.

Thanks,
Fausto


On Wed, Jan 27, 2016 at 1:55 PM, Sam Yaple  wrote:

> On Wed, Jan 27, 2016 at 7:06 PM, Flavio Percoco  wrote:
>>
>> FWIW, the current governance model does not prevent competition. That's
>> not to
>> be understood as we encourage it but rather than there could be services
>> with
>> some level of overlap that are still worth being separate.
>>
>> What Jay is referring to is that regardless the projects do similar
>> things, the
>> same or totally different things, we should strive to have different
>> APIs. The
>> API shouldn't overlap in terms of endpoints and the way they are exposed.
>>
>> With all that said, I'd like to encourage collaboration over competition
>> and I'm
>> sure both teams will find a way to make this work.
>>
>> Cheers,
>> Flavio
>
>
> And to come full circle on this thread, I will point out once again there
> is no competition between Ekko and Freezer at this time. Freezer is
> file-level backup where Ekko is block-level backup. Anyone familiar with
> backups knows these are drastically different types of backups. Those using
> block-level backups typically won't be using file-level backups and
> vice-versa. That said, even if there is no convergence of Freezer and Ekko
>  they can still live side-by-side without any conflict at all.
>
> As of now, Ekko and Freezer teams have started a dialogue and we will
> continue to collaborate rather than compete in every way that is reasonable
> for both projects.
>
> Sam Yaple
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[gordon chung]

> It makes for a crappy user experience. Crappier than the crappy user
> experience that OpenStack API users already have because we have done a
> crappy job shepherding projects in order to make sure there isn't
> overlap between their APIs (yes, Ceilometer and Monasca, I'm looking
> directly at you).
... yes, Ceilometer can easily handle your events and meters and store 
them in either Elasticsearch or Gnocchi for visualisations. you just 
need to create a new definition in our mapping files[1][2]. you will 
definitely want to coordinate the naming of your messages. ie. 
event_type == backup. and event_type == backup..

[1] 
https://github.com/openstack/ceilometer/blob/master/etc/ceilometer/event_definitions.yaml
[2] 
https://github.com/openstack/ceilometer/blob/master/ceilometer/meter/data/meters.yaml

cheers,

-- 
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Jay Pipes]

On 01/26/2016 02:47 AM, Sam Yaple wrote:

Hello Fausto,

I am happy to have a conversation about this with you and the Freezer
team. I have a feeling the current direction of Ekko will add many
components that will not be needed for Freezer and vice-versa.
Nevertheless, I am all about community!


My personal request is that the two contributor communities do 
everything in their power to ensure that the REST API endpoints are not 
overlapping. The last thing we need is to have two APIs for performing 
backups that are virtually identical to each other.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Tue, Jan 26, 2016 at 10:15 AM, Jay Pipes  wrote:

> On 01/26/2016 02:47 AM, Sam Yaple wrote:
>
>> Hello Fausto,
>>
>> I am happy to have a conversation about this with you and the Freezer
>> team. I have a feeling the current direction of Ekko will add many
>> components that will not be needed for Freezer and vice-versa.
>> Nevertheless, I am all about community!
>>
>
> My personal request is that the two contributor communities do everything
> in their power to ensure that the REST API endpoints are not overlapping.
> The last thing we need is to have two APIs for performing backups that are
> virtually identical to each other.
>
>
The way I see this situation is the same as asking Ekko to integrate with
cinder-backup because they perform backups that are "virtually identical"
to each other. They aren't actually related at all, other than perhaps an
API call that says 'backup'. Actual implementation and end results are
wildly different. So my question would be, how would you go about solving
that situation? I could absolutely get on board with sharing an API and
even scheduler, but Ekko and Freezer are two distinct projects solving
different issues with different infrastructure requirements and I am not
aware of anyway to share APIs between projects other than merging the
projects.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Dean Troyer]

On Tue, Jan 26, 2016 at 9:28 AM, Sam Yaple  wrote:

> On Tue, Jan 26, 2016 at 10:15 AM, Jay Pipes  wrote:
>
>> My personal request is that the two contributor communities do everything
>> in their power to ensure that the REST API endpoints are not overlapping.
>> The last thing we need is to have two APIs for performing backups that are
>> virtually identical to each other.
>>
>>
> The way I see this situation is the same as asking Ekko to integrate with
> cinder-backup because they perform backups that are "virtually identical"
> to each other. They aren't actually related at all, other than perhaps an
> API
>

But you see this is exactly where they are directly related to everyone who
is not a developer of the back-end services.  Everything that wants to do a
volume backup (users, other services, etc) should not have multiple choices
to perform that backup, irregardless of how that action is implemented.


> call that says 'backup'. Actual implementation and end results are wildly
> different. So my question would be, how would you go about solving that
> situation? I could absolutely get on board with sharing an API and even
> scheduler, but Ekko and Freezer are two distinct projects solving different
> issues with different infrastructure requirements and I am not aware of
> anyway to share APIs between projects other than merging the projects.
>

Perhaps this is a problem whose time has come to address?


I want to expand a bit on Jay's overlapping API comment.  It is at the
beginning of a project like this that we have the one and only chance of
getting the API right without having to worry about backward compatibility.

In the field today we have a large number of consumers of our APIs
(Horizon, our own client libs, the Python SDK, jclouds, libcloud,
gophercloud, and the list goes on) not to mention those who are writing
their own for various reasons.  Making sense of these across projects gets
more important with every new project that intends to become 'one of us'.

We (OpenStack community, from day 2) have always considered back-end
implementation as one of the major differentaiting factors in our
projects.  Our API consumers frankly don't care about that.  We present a
lot of endpoints and APIs.  But for those areas where there may be some
overlap, or for even competing implementations, we should be working from
the beginning to make these APIs look sensible to those who consume them.

It just so happens that the API working group is addressing some of these
things related to the API structure and striving for consistency across
OpenStack.  However, that is more of a matter of form and structure, not of
implementation and back-end structure.

I would hate to see us keep duplicating user-facing stuff for the sake of
back-end developer convenience.

dt

-- 

Dean Troyer
dtro...@gmail.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

Didn't hit the mailing list with the last reply. Forwarding to a wider
audience than just Dean
-- Forwarded message --
From: "Sam Yaple" <sam...@yaple.net>
Date: Jan 26, 2016 12:00 PM
Subject: Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup
for OpenStack
To: "Dean Troyer" <dtro...@gmail.com>
Cc:


On Jan 26, 2016 11:42 AM, "Dean Troyer" <dtro...@gmail.com> wrote:
>
> On Tue, Jan 26, 2016 at 9:28 AM, Sam Yaple <sam...@yaple.net> wrote:
>>
>> On Tue, Jan 26, 2016 at 10:15 AM, Jay Pipes <jaypi...@gmail.com> wrote:
>>>
>>> My personal request is that the two contributor communities do
everything in their power to ensure that the REST API endpoints are not
overlapping. The last thing we need is to have two APIs for performing
backups that are virtually identical to each other.
>>>
>>
>> The way I see this situation is the same as asking Ekko to integrate
with cinder-backup because they perform backups that are "virtually
identical" to each other. They aren't actually related at all, other than
perhaps an API
>
>
> But you see this is exactly where they are directly related to everyone
who is not a developer of the back-end services.  Everything that wants to
do a volume backup (users, other services, etc) should not have multiple
choices to perform that backup, irregardless of how that action is
implemented.

You skipped over the important part. "Actual implementation and end results
are wildly different." They are not the same even if the API call is named
similar, as I originally stated.

> Perhaps this is a problem whose time has come to address?

Absolutely! I would love to not have to worry about building and
maintaining an API. That said, Ekko isn't here to solve that issue.

> I would hate to see us keep duplicating user-facing stuff for the sake of
back-end developer convenience

I agree about not duplicating user-facing things. But it is a bit more than
"back-end developer convenience". I am both an operator and a developer, so
I can speak from both of those points of view when I say I do not want to
be forced to deploy services that I won't use for a feature unrelated to
them. For sale of example, requiring Ceilometer to use Cinder would be
awful. Those wanting to use Ekko may have no interest in using Freezer and
vice versa. Forcing unrelated processes and services for the sake of one
API is not something I agree with.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Fausto Marzi]

Hi Jay, Dean,
totally agree, with Sam, we'll make sure there will not be any overlap.

Sam,
Thanks for your openness. I'd like to have a conversation about it. When
you say "current direction of Ekko will add many components" do you have
any reference, plan or road map where we can see that direction and
components more in detail?

Very soon we'll starting working also on the block based incremental, that
will be overlapping... if Ekko would fit and we do not have to re implement
it, that'd be fantastic, and not overlapping =). I see many points we can
converge.

>From our perspective, we try to focus more on what can be converged rather
then what cannot. Converging is something that allow us to move forward
faster, delivering something better to the Community, that's the only
motivation.

Let's talk on what we can do, I'm totally confident we'll find a way to do
something useful for everybody : ), beside we matured some experience
dealing positively with the OS community, that will benefit you, believe me
: )

Sounds good?

I'll set up a public meeting next week to discuss this.

Many thanks,
Fausto


On Tue, Jan 26, 2016 at 11:03 AM, Sam Yaple <sam...@yaple.net> wrote:

> Didn't hit the mailing list with the last reply. Forwarding to a wider
> audience than just Dean
> -- Forwarded message --
> From: "Sam Yaple" <sam...@yaple.net>
> Date: Jan 26, 2016 12:00 PM
> Subject: Re: [openstack-dev] Announcing Ekko -- Scalable block-based
> backup for OpenStack
> To: "Dean Troyer" <dtro...@gmail.com>
> Cc:
>
>
> On Jan 26, 2016 11:42 AM, "Dean Troyer" <dtro...@gmail.com> wrote:
> >
> > On Tue, Jan 26, 2016 at 9:28 AM, Sam Yaple <sam...@yaple.net> wrote:
> >>
> >> On Tue, Jan 26, 2016 at 10:15 AM, Jay Pipes <jaypi...@gmail.com> wrote:
> >>>
> >>> My personal request is that the two contributor communities do
> everything in their power to ensure that the REST API endpoints are not
> overlapping. The last thing we need is to have two APIs for performing
> backups that are virtually identical to each other.
> >>>
> >>
> >> The way I see this situation is the same as asking Ekko to integrate
> with cinder-backup because they perform backups that are "virtually
> identical" to each other. They aren't actually related at all, other than
> perhaps an API
> >
> >
> > But you see this is exactly where they are directly related to everyone
> who is not a developer of the back-end services.  Everything that wants to
> do a volume backup (users, other services, etc) should not have multiple
> choices to perform that backup, irregardless of how that action is
> implemented.
>
> You skipped over the important part. "Actual implementation and end
> results are wildly different." They are not the same even if the API call
> is named similar, as I originally stated.
>
> > Perhaps this is a problem whose time has come to address?
>
> Absolutely! I would love to not have to worry about building and
> maintaining an API. That said, Ekko isn't here to solve that issue.
>
> > I would hate to see us keep duplicating user-facing stuff for the sake
> of back-end developer convenience
>
> I agree about not duplicating user-facing things. But it is a bit more
> than "back-end developer convenience". I am both an operator and a
> developer, so I can speak from both of those points of view when I say I do
> not want to be forced to deploy services that I won't use for a feature
> unrelated to them. For sale of example, requiring Ceilometer to use Cinder
> would be awful. Those wanting to use Ekko may have no interest in using
> Freezer and vice versa. Forcing unrelated processes and services for the
> sake of one API is not something I agree with.
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Jay Pipes]

On 01/26/2016 03:28 PM, Sam Yaple wrote:

On Tue, Jan 26, 2016 at 10:15 AM, Jay Pipes > wrote:

On 01/26/2016 02:47 AM, Sam Yaple wrote:

Hello Fausto,

I am happy to have a conversation about this with you and the
Freezer
team. I have a feeling the current direction of Ekko will add many
components that will not be needed for Freezer and vice-versa.
Nevertheless, I am all about community!


My personal request is that the two contributor communities do
everything in their power to ensure that the REST API endpoints are
not overlapping. The last thing we need is to have two APIs for
performing backups that are virtually identical to each other.


The way I see this situation is the same as asking Ekko to integrate
with cinder-backup because they perform backups that are "virtually
identical" to each other.


No. You have entirely misunderstood my point.

> They aren't actually related at all, other

than perhaps an API call that says 'backup'. Actual implementation and
end results are wildly different. So my question would be, how would you
go about solving that situation? I could absolutely get on board with
sharing an API and even scheduler, but Ekko and Freezer are two distinct
projects solving different issues with different infrastructure
requirements and I am not aware of anyway to share APIs between projects
other than merging the projects.


I am not suggesting you "share an API" at all. I am requesting that if 
you have a RESTful API planned for your "backup", then you do not use 
the same RESTful API resource endpoint names that Freezer does. Because 
if you do, then users of the OpenStack APIs will have two APIs that use 
identical resource endpoints for entirely different things. So the 
request is to not use Freezer's resource endpoints, which have /backups 
as its primary resource endpoint.


I don't like the fact that Freezer's resource endpoint is /backups, 
since the OpenStack Volume API has a /{tenant_id}/backups resource 
endpoint, but I really, *really* do not want to see a set of OpenStack 
APIs one of which has /{tenant_id}/backups as a resource endpoint, 
another which has /backups as a top-level resource, and still another 
which has /backups as a top-level resource.


It makes for a crappy user experience. Crappier than the crappy user 
experience that OpenStack API users already have because we have done a 
crappy job shepherding projects in order to make sure there isn't 
overlap between their APIs (yes, Ceilometer and Monasca, I'm looking 
directly at you).


Thanks,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Tue, Jan 26, 2016 at 9:57 PM, Jay Pipes  wrote:

>
> I am not suggesting you "share an API" at all. I am requesting that if you
> have a RESTful API planned for your "backup", then you do not use the same
> RESTful API resource endpoint names that Freezer does. Because if you do,
> then users of the OpenStack APIs will have two APIs that use identical
> resource endpoints for entirely different things. So the request is to not
> use Freezer's resource endpoints, which have /backups as its primary
> resource endpoint.
>
> I don't like the fact that Freezer's resource endpoint is /backups, since
> the OpenStack Volume API has a /{tenant_id}/backups resource endpoint, but
> I really, *really* do not want to see a set of OpenStack APIs one of which
> has /{tenant_id}/backups as a resource endpoint, another which has /backups
> as a top-level resource, and still another which has /backups as a
> top-level resource.
>
> It makes for a crappy user experience. Crappier than the crappy user
> experience that OpenStack API users already have because we have done a
> crappy job shepherding projects in order to make sure there isn't overlap
> between their APIs (yes, Ceilometer and Monasca, I'm looking directly at
> you).
>
>
That is a much, much clearer point. One that I will be happy to follow. I
understand and agree with what you are saying.

A more detailed conversation has been scheduled to determine if Ekko and
Freezer can co-exist together sharing resources in a plugin-type fashion.
It is not known if this is possible yet, but if it is not I will certainly
follow your suggestion, Jay. Thank you for your insight!

SamYaple
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

On Mon, Jan 25, 2016 at 8:45 AM, Thierry Carrez 
wrote:

> Sam Yaple wrote:
>
>> We would like to introduce you to a new community-driven OpenStack
>> project called Ekko.
>>
>> The aim of Ekko is to provide incremental block-level backup and restore
>> of Nova instances. We see backups as a key area that is missing in
>> OpenStack. One issue that has previously prevented backups in OpenStack
>> is the scalability of the storage backend. Object-storage is the answer
>> to this scalability problem, but with block-based backups you often see
>> large files that require POSIX operations to perform retention and
>> deletions. These operations are not able to be performed in the
>> traditional way in object storage, which has prevented leveraging
>> object-storage to its full potential. With Ekko we can solve this issue
>> allowing us to use storage that can scale with OpenStack.
>>
>
> Hi!
>
> How does Ekko compare to / differ from Freezer, which is an official
> OpenStack project targeted to the same problem space ? I suspect this is
> more low-level ? Is there some potential for convergence between the two
> projects ?
>
>
Hello Thierry,

These are good questions. The biggest difference you already caught onto,
Ekko is more low-level. Freezer is targeted at the filesystem and specific
applications (like databases) directly.

There are only two places with overlapping goals that I know of*. The first
is backup of a Cinder volume which is a future goal of Ekko and something
Freezer can currently do for LVM backed Cinder. The second is backup of a
nova instance. This isn't something freezer does directly, instead it
leverages nova-snapshot which is very disruptive to the instance and will
cause downtime for said instance. The current pursuit of Ekko is _live_
incremental block-level backup of an nova instance and in that regard there
is no overlap with Freezer or any other project for that matter.

To answer the question of convergence between Ekko and Freezer, I would say
it's possible. That being said both projects are addressing different
problems in different ways. As discussed above, there is little overlap
between the two projects and the areas where there is overlap of goals have
drastically different implementations. I could put together a list of Ekko
vs Freezer, but I think that would be comparing apples to oranges. To state
this in terms of compatibility, Ekko and Freezer can run side-by-side
without interfering with each other in anyway.

*Disclaimer: I am no expert on Freezer, I may be wrong in some statements
and am open to correction.

Sam Yaple

-- 
> Thierry Carrez (ttx)
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Thierry Carrez]

Sam Yaple wrote:

We would like to introduce you to a new community-driven OpenStack
project called Ekko.

The aim of Ekko is to provide incremental block-level backup and restore
of Nova instances. We see backups as a key area that is missing in
OpenStack. One issue that has previously prevented backups in OpenStack
is the scalability of the storage backend. Object-storage is the answer
to this scalability problem, but with block-based backups you often see
large files that require POSIX operations to perform retention and
deletions. These operations are not able to be performed in the
traditional way in object storage, which has prevented leveraging
object-storage to its full potential. With Ekko we can solve this issue
allowing us to use storage that can scale with OpenStack.


Hi!

How does Ekko compare to / differ from Freezer, which is an official 
OpenStack project targeted to the same problem space ? I suspect this is 
more low-level ? Is there some potential for convergence between the two 
projects ?


--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Fausto Marzi]

Hi Sam,
My opinion would be to converge, so to have Ekko features exported from the 
freezer-api and horizon web interface. Also the freezer-scheduler can be 
integrated, that would enable Ekko to execute backup syncronized over multiple 
nodes.

By all mean, this does not mean you have to, it's just how I feel about it.

We are totally open, so please let us know if there's any interest from your 
side.

Thanks,
Fausto

Sent from my iPhone

> On 25 Jan 2016, at 08:58, Sam Yaple  wrote:
> 
>> On Mon, Jan 25, 2016 at 8:45 AM, Thierry Carrez  
>> wrote:
>> Sam Yaple wrote:
>>> We would like to introduce you to a new community-driven OpenStack
>>> project called Ekko.
>>> 
>>> The aim of Ekko is to provide incremental block-level backup and restore
>>> of Nova instances. We see backups as a key area that is missing in
>>> OpenStack. One issue that has previously prevented backups in OpenStack
>>> is the scalability of the storage backend. Object-storage is the answer
>>> to this scalability problem, but with block-based backups you often see
>>> large files that require POSIX operations to perform retention and
>>> deletions. These operations are not able to be performed in the
>>> traditional way in object storage, which has prevented leveraging
>>> object-storage to its full potential. With Ekko we can solve this issue
>>> allowing us to use storage that can scale with OpenStack.
>> 
>> Hi!
>> 
>> How does Ekko compare to / differ from Freezer, which is an official 
>> OpenStack project targeted to the same problem space ? I suspect this is 
>> more low-level ? Is there some potential for convergence between the two 
>> projects ?
> 
> Hello Thierry,
> 
> These are good questions. The biggest difference you already caught onto, 
> Ekko is more low-level. Freezer is targeted at the filesystem and specific 
> applications (like databases) directly.
> 
> There are only two places with overlapping goals that I know of*. The first 
> is backup of a Cinder volume which is a future goal of Ekko and something 
> Freezer can currently do for LVM backed Cinder. The second is backup of a 
> nova instance. This isn't something freezer does directly, instead it 
> leverages nova-snapshot which is very disruptive to the instance and will 
> cause downtime for said instance. The current pursuit of Ekko is _live_ 
> incremental block-level backup of an nova instance and in that regard there 
> is no overlap with Freezer or any other project for that matter.
> 
> To answer the question of convergence between Ekko and Freezer, I would say 
> it's possible. That being said both projects are addressing different 
> problems in different ways. As discussed above, there is little overlap 
> between the two projects and the areas where there is overlap of goals have 
> drastically different implementations. I could put together a list of Ekko vs 
> Freezer, but I think that would be comparing apples to oranges. To state this 
> in terms of compatibility, Ekko and Freezer can run side-by-side without 
> interfering with each other in anyway.
> 
> *Disclaimer: I am no expert on Freezer, I may be wrong in some statements and 
> am open to correction.
> 
> Sam Yaple
> 
>> -- 
>> Thierry Carrez (ttx)
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

Hello Fausto,

I am happy to have a conversation about this with you and the Freezer team.
I have a feeling the current direction of Ekko will add many components
that will not be needed for Freezer and vice-versa. Nevertheless, I am all
about community!

Sam Yaple

On Tue, Jan 26, 2016 at 2:20 AM, Fausto Marzi 
wrote:

> Hi Sam,
> My opinion would be to converge, so to have Ekko features exported from
> the freezer-api and horizon web interface. Also the freezer-scheduler can
> be integrated, that would enable Ekko to execute backup syncronized over
> multiple nodes.
>
> By all mean, this does not mean you have to, it's just how I feel about it.
>
> We are totally open, so please let us know if there's any interest from
> your side.
>
> Thanks,
> Fausto
>
> Sent from my iPhone
>
> On 25 Jan 2016, at 08:58, Sam Yaple  wrote:
>
> On Mon, Jan 25, 2016 at 8:45 AM, Thierry Carrez 
> wrote:
>
>> Sam Yaple wrote:
>>
>>> We would like to introduce you to a new community-driven OpenStack
>>> project called Ekko.
>>>
>>> The aim of Ekko is to provide incremental block-level backup and restore
>>> of Nova instances. We see backups as a key area that is missing in
>>> OpenStack. One issue that has previously prevented backups in OpenStack
>>> is the scalability of the storage backend. Object-storage is the answer
>>> to this scalability problem, but with block-based backups you often see
>>> large files that require POSIX operations to perform retention and
>>> deletions. These operations are not able to be performed in the
>>> traditional way in object storage, which has prevented leveraging
>>> object-storage to its full potential. With Ekko we can solve this issue
>>> allowing us to use storage that can scale with OpenStack.
>>>
>>
>> Hi!
>>
>> How does Ekko compare to / differ from Freezer, which is an official
>> OpenStack project targeted to the same problem space ? I suspect this is
>> more low-level ? Is there some potential for convergence between the two
>> projects ?
>>
>>
> Hello Thierry,
>
> These are good questions. The biggest difference you already caught onto,
> Ekko is more low-level. Freezer is targeted at the filesystem and specific
> applications (like databases) directly.
>
> There are only two places with overlapping goals that I know of*. The
> first is backup of a Cinder volume which is a future goal of Ekko and
> something Freezer can currently do for LVM backed Cinder. The second is
> backup of a nova instance. This isn't something freezer does directly,
> instead it leverages nova-snapshot which is very disruptive to the instance
> and will cause downtime for said instance. The current pursuit of Ekko is
> _live_ incremental block-level backup of an nova instance and in that
> regard there is no overlap with Freezer or any other project for that
> matter.
>
> To answer the question of convergence between Ekko and Freezer, I would
> say it's possible. That being said both projects are addressing different
> problems in different ways. As discussed above, there is little overlap
> between the two projects and the areas where there is overlap of goals have
> drastically different implementations. I could put together a list of Ekko
> vs Freezer, but I think that would be comparing apples to oranges. To state
> this in terms of compatibility, Ekko and Freezer can run side-by-side
> without interfering with each other in anyway.
>
> *Disclaimer: I am no expert on Freezer, I may be wrong in some statements
> and am open to correction.
>
> Sam Yaple
>
> --
>> Thierry Carrez (ttx)
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Announcing Ekko -- Scalable block-based backup for OpenStack

[Sam Yaple]

Dear community,

We would like to introduce you to a new community-driven OpenStack project
called Ekko.

The aim of Ekko is to provide incremental block-level backup and restore of
Nova instances. We see backups as a key area that is missing in OpenStack.
One issue that has previously prevented backups in OpenStack is the
scalability of the storage backend. Object-storage is the answer to this
scalability problem, but with block-based backups you often see large files
that require POSIX operations to perform retention and deletions. These
operations are not able to be performed in the traditional way in object
storage, which has prevented leveraging object-storage to its full
potential. With Ekko we can solve this issue allowing us to use storage
that can scale with OpenStack.

Based on previous projects [1] and research that has been progressing for
the past 2 years or so, Ekko is being completely written from scratch by
the community, incorporating these learnings. This research combined with a
new feature available in QEMU 2.4 [2] will allow us to bring incremental
block-based backups to OpenStack.

The project is in it's beginning stages, however, we already have active
developers submitting code and collaborating via IRC and code review [3].
We welcome thoughts, questions, and feedback. You can join us on
#openstack-ekko, and be sure to keep an eye out for our talk at the summit
to learn more.

Best Regards,
The Ekko Team

[1] https://github.com/SamYaple/osdk
[2] http://wiki.qemu.org/Features/IncrementalBackup
[3] https://review.openstack.org/#/q/project:openstack/ekko
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev