Re: [openstack-dev] Split of the openstack-dev list

[Daniel P. Berrange]

On Thu, Nov 14, 2013 at 02:19:24PM +0100, Thierry Carrez wrote:
> Thierry Carrez wrote:
> > [...]
> > That will not solve all issues. We should also collectively make sure
> > that *usage questions are re-routed* to the openstack general
> > mailing-list, where they belong. Too many people still answer off-topic
> > questions here on openstack-dev, which encourages people to be off-topic
> > in the future (traffic on the openstack general ML has been mostly
> > stable, with only 868 posts in October). With those actions, I hope that
> > traffic on openstack-dev would drop back to the 1000-1500 range, which
> > would be more manageable for everyone.
> 
> Other suggestion: we could stop posting meeting reminders to -dev (I
> know, I'm guilty of it) and only post something if the meeting time
> changes, or if the weekly meeting is canceled for whatever reason.

Is there somewhere on the website which keeps a record of all regular
scheduled meetings people can discover / refer to easily ?

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [reviews] putting numbers on -core team load

[Daniel P. Berrange]

On Thu, Nov 14, 2013 at 08:29:36PM +1300, Robert Collins wrote:
> At the summit there were lots of discussions about reviews, and I made
> the mistake of sending a mail to Russell proposing a few new stats we
> could gather.
> 
> I say mistake, because he did so and then some... we new have extra
> info - consider:
> 
> http://russellbryant.net/openstack-stats/nova-reviewers-30.txt
> 
> There are two new things:
> In each row a new column 'received' - this counts the number of
> incoming reviews to each reviewer. It's use should be obvious - but
> remember that folk who contribute the occasional patch probably don't
> have the context to be doing reviews... those who are contributing
> many patches and getting many incoming reviews however...
> This gives us the philanthropists (or perhaps team supporters...)
> 
> |klmitch **   | 1370  19   0 118   986.1% |
> 14 ( 11.9%)  |  0 (∞)|
> 
> (the bit at the end is a unicode infinity... we'll need to work on that).
> 
> And so on :)
> 
> Down the bottom of the page:
> 
> Total reviews: 2980 (99.3/day)
> Total reviewers: 241
> Total reviews by core team: 1261 (42.0/day)
> Core team size: 17
> New patch sets in the last 30 days: 1989 (66.3/day)
> 
> and for 90 days:
> 
> Total reviews: 10705 (118.9/day)
> Total reviewers: 406
> Total reviews by core team: 5289 (58.8/day)
> Core team size: 17
> New patch sets in the last 90 days: 7515 (83.5/day)
> 
> This is the really interesting bit. Remembering that every patch needs
> - at minimum - 2 +2's, the *minimum* viable core team review rate to
> keep up is patch sets per day * 2:
> 30 days: 132 core reviews/day
> 90 days: 167 core reviews/day
> 
> But we're getting:
> 30 days: 42/day or 90/day short
> 90 days: 59/day or 108/day short
> 
> One confounding factor here is that this counts (AIUI) pushed changes,
> not change ids - so we don't need two +2's for every push, we need two
> +2's for every changeid - we should add that as a separate metric I
> think, as the needed +2 count will be a bit lower.

NB, this analysis seems to assume that we actally /want/ to eventually
approve ever submitted patch. There's going to be some portion that are
abandoned or rejected. It is probably a reasonably small portion of the
total so won't change the order of magnitude, but I'd be interested in
stats on how many patches we reject in one way or another.

> Anyhow, to me - this is circling in nicely on having excellent
> information (vs data) on the review status, and from there we can
> start to say 'right, to keep up, Nova needs N core reviewers
> consistently doing Y reviews per day. If Y is something sensible like
> 3 or 4, we can work backwards. Using the current figures (which since
> we don't have changeId as a separate count are a little confounded)
> that would give us:
> 
> time period  reviews/core/day core-team-size
> 30 days   344
> 30 days   433
> 30 days   817
> 90 days   356
> 90 days   442
> 90 days   10  17
> 
> Also note that these are calendar days, so no weekends or leave for -core!
> 
> What else
> 
> in the last 30 days core have done 42% of reviews, in the last 90 days
> 49%. So thats getting better.
> 
> I know Russell has had concerns about core cohesion in the past, but I
> don't think doing 8 detailed reviews every day including weekends is
> individually sustainable. IMO we badly need more core reviewers
> and that means:
> 
>  - 20 or so volunteers
>  - who step up and do - pick a number - say 3 - reviews a day, every
> work day, like clockwork.
>  - and follow up on their reviewed patches to learn what other
> reviewers say, and why
>  - until the nova-core team & Russell are happy that they can
> contribute effectively as -core.
> 
> Why such a big number of volunteers? Because we need a big number of
> people to spread load, because Nova has a high incoming patch rate.

One concern I have is that this exercise could turn out to be a bit like
building new roads to solve traffic jams. The more roads you build, the
more car usage you trigger. In some ways the limited rate of approvals
can be seen as acting as a natural break on the rate of patch submissions
getting out of control. I'm not saying we've reached that point currently,
but at some point I think it is wise to ask what is the acceptable rate
of code churn we should try to support as a project.


One of the problems of a project as large as Nova is that it is hard for
one person to be expert at reviewing all areas of the code. As you increase
the size of core review team we have to be careful we don't get a situation
where we have too many patches being reviewed & approved by people who are
not the natural experts in an area. At the same time you don't want people
to be strictly siloed to

Re: [openstack-dev] Introducing the new OpenStack service for Containers

[Daniel P. Berrange]

On Mon, Nov 18, 2013 at 07:10:53PM -0500, Krishna Raman wrote:
> We should probably meet on IRC or conf. call and discuss the suggested 
> architecture, open questions and REST API.
> 
> I have set up a doodle poll at http://doodle.com/w7y5qcdvq9i36757 to gather a 
> times when we can meet.
> Please sign up or let me know if none of the time slots works for you.

The time slots are pretty west coast centric there. Only the 8am/9am slots
work for people in Europe really.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Introducing the new OpenStack service for Containers

[Daniel P. Berrange]

On Tue, Nov 19, 2013 at 10:57:51AM +0100, Thierry Carrez wrote:
> Russell Bryant wrote:
> > My view of the outcome of the session was not "it *will* be a new
> > service".  Instead, it was, "we *think* it should be a new service, but
> > let's do some more investigation to decide for sure".
> > 
> > The action item from the session was to go off and come up with a
> > proposal for what a new service would look like.  In particular, we
> > needed a proposal for what the API would look like.  With that in hand,
> > we need to come back and ask the question again of whether a new service
> > is the right answer.
> 
> +1
> 
> I can see how a separate service can be a good way to avoid making Nova
> support container-specific use cases and make it even bigger than it is.
> However if you end up duplicating most of the code, I'm not sure there
> would be any net gain.
> 
> I'm not talking about the base service infrastructure and the scheduler
> (which are well-known duplication already) but more around specific nova
> features (metadata/config drive, networking, image handling, etc.) that
> would apply to VMs and containers alike.
> 
> So it would be great to come out of this first round of analysis with a
> plan detailing how different (and how similar) from nova that new
> service would be, and how much code duplication is to be expected.

Yep, I'm far from convinced that having a separate service for
containers is going to be a net gain for the project as a whole.
It seems to me that we're likely to have an API and codebase that
is 95% the same in both cases here, with most differences just
being about the way things are initially booted/provisioned.

While containers certainly have some unique attributes, I believe
the level of differentiation is overblown. I also think the difference
is not about  containers vs VMs, but rather about OS virtualization vs
application virtualization.

It is entirely practical to do application virtualization using
either containers or VMs, as demonstrated by libvirt-sandbox which
can run individual app processes inside KVM without needing a full
OS intsall for it. There are notable security advantages to using
KVM for application virtualization since it avoids the shared kernel
single point of failure/compromise.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Blueprint: standard specification of guest CPU topology

[Daniel P. Berrange]

For attention of maintainers of Nova virt drivers

A while back there was a bug requesting the ability to set the CPU
topology (sockets/cores/threads) for guests explicitly

   https://bugs.launchpad.net/nova/+bug/1199019

I countered that setting explicit topology doesn't play well with
booting images with a variety of flavours with differing vCPU counts.

This led to the following change which used an image property to
express maximum constraints on CPU topology (max-sockets/max-cores/
max-threads) which the libvirt driver will use to figure out the
actual topology (sockets/cores/threads)

  https://review.openstack.org/#/c/56510/

I believe this is a prime example of something we must co-ordinate
across virt drivers to maximise happiness of our users.

There's a blueprint but I find the description rather hard to
follow

  https://blueprints.launchpad.net/nova/+spec/support-libvirt-vcpu-topology

So I've created a standalone wiki page which I hope describes the
idea more clearly

  https://wiki.openstack.org/wiki/VirtDriverGuestCPUTopology

Launchpad doesn't let me link the URL to the blueprint since I'm not
the blurprint creator :-(

Anyway this mail is to solicit input on the proposed standard way to
express this which is hypervisor portable and the addition of some
shared code for doing the calculations which virt driver impls can
just all into rather than re-inventing

I'm looking for buy-in to the idea from the maintainers of each
virt driver that this conceptual approach works for them, before
we go merging anything with the specific impl for libvirt.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Core pinning

[Daniel P. Berrange]

On Wed, Nov 13, 2013 at 02:46:06PM +0200, Tuomas Paappanen wrote:
> Hi all,
> 
> I would like to hear your thoughts about core pinning in Openstack.
> Currently nova(with qemu-kvm) supports usage of cpu set of PCPUs
> what can be used by instances. I didn't find blueprint, but I think
> this feature is for isolate cpus used by host from cpus used by
> instances(VCPUs).
> 
> But, from performance point of view it is better to exclusively
> dedicate PCPUs for VCPUs and emulator. In some cases you may want to
> guarantee that only one instance(and its VCPUs) is using certain
> PCPUs.  By using core pinning you can optimize instance performance
> based on e.g. cache sharing, NUMA topology, interrupt handling, pci
> pass through(SR-IOV) in multi socket hosts etc.
> 
> We have already implemented feature like this(PoC with limitations)
> to Nova Grizzly version and would like to hear your opinion about
> it.
> 
> The current implementation consists of three main parts:
> - Definition of pcpu-vcpu maps for instances and instance spawning
> - (optional) Compute resource and capability advertising including
> free pcpus and NUMA topology.
> - (optional) Scheduling based on free cpus and NUMA topology.
> 
> The implementation is quite simple:
> 
> (additional/optional parts)
> Nova-computes are advertising free pcpus and NUMA topology in same
> manner than host capabilities. Instances are scheduled based on this
> information.
> 
> (core pinning)
> admin can set PCPUs for VCPUs and for emulator process, or select
> NUMA cell for instance vcpus, by adding key:value pairs to flavor's
> extra specs.
> 
> EXAMPLE:
> instance has 4 vcpus
> :
> vcpus:1,2,3,4 --> vcpu0 pinned to pcpu1, vcpu1 pinned to pcpu2...
> emulator:5 --> emulator pinned to pcpu5
> or
> numacell:0 --> all vcpus are pinned to pcpus in numa cell 0.
> 
> In nova-compute, core pinning information is read from extra specs
> and added to domain xml same way as cpu quota values(cputune).
> 
> 
>   
>   
>   
>   
>   
> 
> 
> What do you think? Implementation alternatives? Is this worth of
> blueprint? All related comments are welcome!

I think there are several use cases mixed up in your descriptions
here which should likely be considered independantly

 - pCPU/vCPU pinning

   I don't really think this is a good idea as a general purpose
   feature in its own right. It tends to lead to fairly inefficient
   use of CPU resources when you consider that a large % of guests
   will be mostly idle most of the time. It has a fairly high
   administrative burden to maintain explicit pinning too. This
   feels like a data center virt use case rather than cloud use
   case really.

 - Dedicated CPU reservation

   The ability of an end user to request that their VM (or their
   group of VMs) gets assigned a dedicated host CPU set to run on.
   This is obviously something that would have to be controlled
   at a flavour level, and in a commercial deployment would carry
   a hefty pricing premium.

   I don't think you want to expose explicit pCPU/vCPU placement
   for this though. Just request the high level concept and allow
   the virt host to decide actual placement

 - Host NUMA placement.

   By not taking NUMA into account currently the libvirt driver
   at least is badly wasting resources. Having too much cross-numa
   node memory access by guests just kills scalability. The virt
   driver should really automaticall figure out cpu & memory pinning
   within the scope of a NUMA node automatically. No admin config
   should be required for this.

 - Guest NUMA topology

   If the flavour memory size / cpu count exceeds the size of a
   single NUMA node, then the flavour should likely have a way to
   express that the guest should see multiple NUMA nodes. The
   virt host would then set guest NUMA topology to match the way
   it places vCPUs & memory on host NUMA nodes. Again you don't
   want explicit pcpu/vcpu mapping done by the admin for this.



Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Core pinning

[Daniel P. Berrange]

On Wed, Nov 13, 2013 at 11:57:22AM -0600, Chris Friesen wrote:
> On 11/13/2013 11:40 AM, Jiang, Yunhong wrote:
> 
> >>But, from performance point of view it is better to exclusively
> >>dedicate PCPUs for VCPUs and emulator. In some cases you may want
> >>to guarantee that only one instance(and its VCPUs) is using certain
> >>PCPUs.  By using core pinning you can optimize instance performance
> >>based on e.g. cache sharing, NUMA topology, interrupt handling, pci
> >>pass through(SR-IOV) in multi socket hosts etc.
> >
> >My 2 cents. When you talking about " performance point of view", are
> >you talking about guest performance, or overall performance? Pin PCPU
> >is sure to benefit guest performance, but possibly not for overall
> >performance, especially if the vCPU is not consume 100% of the CPU
> >resources.
> 
> It can actually be both.  If a guest has several virtual cores that
> both access the same memory, it can be highly beneficial all around
> if all the memory/cpus for that guest come from a single NUMA node
> on the host.  That way you reduce the cross-NUMA-node memory
> traffic, increasing overall efficiency.  Alternately, if a guest has
> several cores that use lots of memory bandwidth but don't access the
> same data, you might want to ensure that the cores are on different
> NUMA nodes to equalize utilization of the different NUMA nodes.
> 
> Similarly, once you start talking about doing SR-IOV networking I/O
> passthrough into a guest (for SDN/NFV stuff) for optimum efficiency
> it is beneficial to be able to steer interrupts on the physical host
> to the specific cpus on which the guest will be running.  This
> implies some form of pinning.

I would say intelligent NUMA placement is something that virt drivers
should address automatically without any need for admin defined pinning.
The latter is just imposing too much admin burden, for something we can
figure out automatically to a good enough extent.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] When is a blueprint unnecessary?

[Daniel P. Berrange]

On Tue, Nov 19, 2013 at 10:56:10AM -0500, Russell Bryant wrote:
> Greetings,
> 
> One of the bits of feedback that came from the "Nova Project Structure
> and Process" session at the design summit was that it would be nice to
> skip having blueprints for smaller items.
> 
> In an effort to capture this, I updated the blueprint review criteria
> [1] with the following:
> 
>   Some blueprints are closed as unnecessary. Blueprints are used for
>   tracking significant development efforts. In general, small and/or
>   straight forward cleanups do not need blueprints. A blueprint should
>   be filed if:
> 
>- it impacts the release notes
>- it covers significant internal development or refactoring efforts

"Impacts release notes" can seem a bit nebulous. It shifts the problem
from
 
   "what is a blueprint required for?"

to

   "what is a release note required for?"

which can be just as ill-defined for many people. So we might want
to be more explicit to enumerate all the things that would typically
trigger release notes, and thus blueprints, explicitly

 - Adds configuration parameters
 - Adds glance/cinder metdata properties
 - Changes public APIs
 - Affects database schema
   ...probably many other things...

> So, to pick out some examples (not trying to pick on anything in
> particular here):
> 
> An example of a feature that should be in release notes:
>   https://blueprints.launchpad.net/nova/+spec/xenapi-resize-ephemeral-disks
> 
> An example of a major refactoring effort that justifies a blueprint:
>   https://blueprints.launchpad.net/nova/+spec/live-migration-to-conductor
> 
> An example of something I think is small enough to not need a blueprint:
>   https://blueprints.launchpad.net/nova/+spec/no-import-uuid
> 
> 
> Determining when a blueprint is and is not necessary is a bit difficult
> to define clearly.  I would appreciate input on how we can define it
> more cleanly so that it can be enforced by the blueprint reviewers
> consistently.

I'd also suggest another criteria for

 - Impacts cross-project interaction/interfaces

 eg calls between Nova & Neutron for VIF setup
   or betweeen Nova & Cinder for volume setup

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] New hypervisor to manage LXC containers on Nova with Docker

[Daniel P. Berrange]

On Thu, Jun 06, 2013 at 10:33:12AM -0700, Sam Alba wrote:
> Hi all,
> 
> I work for dotCloud and I wanted to share our recent work on Nova[1].
> We've been using LXC containers on our PaaS product for more than 2
> years now and we recently released the Docker project[2] that you may
> have heard about already.
> 
> We also wrote a blog post[3] to explain the work we did and how the
> Glance can make the glue between Nova and Docker's remote Images
> index.

Some people asked offlist what my opinion was on accepting this new
virt driver, given that we already have Libvirt LXC driver support
in Nova.

Superficially this appears to be the same scenario that was recently
raised wrt supporting a new OpenVZ driver in Nova, vs supporting it
via the Libvirt OpenVZ capabilities (where we chose the latter).

"LXC" can mean many things. In particular it can refer to the kernel
features that have been implemented to support containers on Linux
or it can refer to a specific userspace toolset implementation that
uses those kernel features. There is the Libvirt LXC driver or there
is the sf.net LXC toolset providing separate userspace implementations.
IIUC, Docker is an application that is built on top of the sf.net
LXC toolset. Conceptually it looks like it works at a higher level
than the raw LXC toolset, so cannot be considered to duplicate either
the Libvirt LXC / sf.net LXC toolset, but rather to complement them.

As such, I believe this is a different situation to the one that
recently arose wrt OpenVZ. A "Docker" driver for Nova would be
providing integration with different capability, than is provided
for by the Libvirt LXC driver. So I think there could be merit in
having Docker support in Nova, provided the submitter (or another
interested party) is prepared to commit to being an active maintainer
for the code long term, and not just submit it and expect someone
else to maintain it after merge.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Quantum's new name is….

[Daniel P. Berrange]

On Wed, Jun 19, 2013 at 12:14:24PM -0400, Mark McClain wrote:
> All-
> 
> The OpenStack Networking team is happy to announce that the Quantum
> project will be changing its name to Neutron. You'll soon see Neutron
> in lots of places as we work to implement the name change within OpenStack.

Are there details of what things are going to be renamed ? If code is
going to be renamed, do we have plans on how to provide a reliable
upgrade path for all user data and configuration across both Neutron
and things configured to talk to it like Nova.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically

[Daniel P. Berrange]

On Fri, Jun 21, 2013 at 09:10:32AM +, Bob Ball wrote:
> It seems that numad is libvirt specific - is that the case?

No, it is a completely independant project

  https://git.fedorahosted.org/git/numad.git

It existed before libvirt started using it for automatic placement.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically

[Daniel P. Berrange]

On Thu, Jun 20, 2013 at 12:48:16PM -0400, Russell Bryant wrote:
> On 06/20/2013 10:36 AM, Giorgio Franceschi wrote:
> > Hello, I created a blueprint for the implementation of:
> > 
> > A tool for pinning automatically each running virtual CPU to a physical
> > one in the most efficient way, balancing load across sockets/cores and
> > maximizing cache sharing/minimizing cache misses. Ideally able to be run
> > on-demand, as a periodic job, or be triggered by events on the host (vm
> > spawn/destroy).
> > 
> > Find it at https://blueprints.launchpad.net/nova/+spec/auto-cpu-pinning
> > 
> > Any inputappreciated!
> 
> I'm actually surprised to see a new tool for this kind of thing.
> 
> Have you seen numad?

The approach used by 'pinhead' tool dscribed in the blueprint seems
to be pretty much equivalent to what 'numad' is already providing
for Libvirt KVM and LXC guests.

NB, numad is actually a standalone program for optimizing NUMA
placement of any processes on a server. Libvirt talks to it when
starting a guest to request info on where best to place the guest.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] SecurityImpact tagging in gerrit

[Daniel P. Berrange]

On Fri, Jun 21, 2013 at 12:08:43PM -0400, Yun Mao wrote:
> Interesting. Does it automatically make the commit in "stealth mode" so
> that it's not seen in public? Thanks,

This tag is about asking for design input / code review from people with
security expertize for new work. As such the code is all public.

Fixes for security flaws in existing code which need to be kept private
should not be sent via Gerrit. They should be reported privately as per
the guidelines here:

  http://www.openstack.org/projects/openstack-security/

> On Fri, Jun 21, 2013 at 11:26 AM, Bryan D. Payne  wrote:
> 
> > This is a quick note to announce that the OpenStack gerrit system supports
> > a SecurityImpact tag.  If you are familiar with the DocImpact tag, this
> > works in a similar fashion.
> >
> > Please use this in the commit message for any commits that you feel would
> > benefit from a security review.  Commits with this tag in the commit
> > message will automatically trigger an email message to the OpenStack
> > Security Group, allowing you to quickly tap into some of the security
> > expertise in our community.
> >
> > PTLs -- Please help spread the word an encourage use of this within your
> > projects.
> >
> > Cheers,
> > -bryan


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Efficiently pin running VMs to physical CPUs automatically

[Daniel P. Berrange]

On Fri, Jun 21, 2013 at 05:09:24PM +, Bob Ball wrote:
> Sorry, my point about numad being libvirt specific was that I
> couldn't find references to other hypervisors using numad for
> their placement.  I recognise that it's not _tied_ to libvirt
> but the reality seems to be that only libvirt uses it.
> 
> Xen, for example, can't use numad because dom0 might only know
> about a subset of the system - it'd make sense for dom0 to only
> be placed on a single numa node. Xen does of course have its own
> automatic placement to take account of the numa nodes - I assume
> this is also true of other hypervisors.

That is merely a limitation of the current impl, not a technology
roadblock. numad could easily be made to ask the Xen hypervisor
what the topology of the entire host was if that was desired for
Xen.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Termination of the title line of commit messages

[Daniel P. Berrange]

On Mon, Jun 24, 2013 at 10:50:18PM +0100, Mark McLoughlin wrote:
> Hey,
> 
> Pulling this out of gerrit for discussion.
> 
> Background is one of my patches to diskimage-builder was -1ed because I
> terminated the title line of the commit message with a period:
> 
>   https://review.openstack.org/33262
> 
> This is actually the exact opposite to what I consider normal practice
> for git commit messages as I explained in the review and the tripleo
> wiki page, so I proposed a hacking change here:
> 
>   https://review.openstack.org/33789
> 
> The rationale for *not* having a period is:
> 
>   * With the 50 char limit, space is at a premium on the first line
> 
>   * The first line is often used as the Subject: in [PATCH] emails -
> subject lines in emails generally don't end in a period
> 
>   * Examples in:
> 
>   
> https://wiki.openstack.org/wiki/GitCommitMessages#Summary_of_GIT_commit_message_structure
> 
> don't end in period
> 
> (Note - the "should not end with a period" was only added by me 
> recently)
> 
>   * Another common reference on git commit messages
> 
>   http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html 
> doesn't either
> 
>   * In git's own git repo, 1.43% of commit messages in the last year 
> ended in a period
> 
>   * I'm not aware of any other OpenStack project which enforces this. 
> Looking at the history of various projects for the past year (and 
> excluding merge commits which don't end with a period), the use of 
> period termination runs at between 10 and 30%.

100% agreed with all this. Using a period at the end of line is
completely pointless. In absence of any explicit hacking rules
to the contrary, commit messages first line should be treated
in the same way as email subject line. No one ever uses a period
in the email subject line.

> I'm pretty confident that danpb didn't mention this when he wrote the
> page because he either felt it was obvious or not worth mentioning. I've
> cc-ed him to be sure.

Yep, I never even considered that anyone would ever want to use a
period at the end of a commit message first line, so never thought
to explicit tell anyone not to.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Nominating John Garbutt for nova-core

[Daniel P. Berrange]

On Wed, Jun 26, 2013 at 11:09:34AM -0400, Russell Bryant wrote:
> Greetings,
> 
> I would like to nominate John Garbutt for the nova-core team.
> 
> John has been involved with nova for a long time now.  He's primarily
> known for his great work on the xenapi driver.  However, he has been
> contributing and reviewing in other areas, as well.  Based on my
> experience with him I think he would be a good addition, so it would be
> great to have him on board to help keep up with the review load.
> 
> Please respond with +1s or any concerns.

+1

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon] python-selenium is non-free, Horizon shouldn't use it

[Daniel P. Berrange]

On Thu, Jul 04, 2013 at 09:34:01AM -0400, Julie Pichon wrote:
> Hi Sascha,
> 
> "Sascha Peilicke"  wrote:
> > On 07/04/2013 02:34 PM, Sascha Peilicke wrote:
> > > On 07/04/2013 12:03 PM, Matthias Runge wrote:
> > >> On 04/07/13 11:27, Thomas Goirand wrote:
> > >>> Horizon seems to use python-selenium. The problem is that, in Debian,
> > >>> this package is in the non-free repository. So I strongly suggest to not
> > >>> use it for Havana. That otherwise would put Horizon into the contrib
> > >>> repository of Debian (eg: not officially in Debian), or eventually,
> > >>> remove any possibility to run the unit tests, which isn't nice.
> > >>>
> > >> Thank you for the heads-up.
> > >>
> > >> Selenium is used for tests during development, it is not a runtime
> > >> requirement at all.
> > >>
> > >> Would that still make it non-free for Debian?
> > >
> > > BTW. this is identical for openSUSE.
> > >
> > >> How did Horizon went into Debian packages at all, since the situation in
> > >> this front is unchanged for at least a year (just curious).
> > >
> > > At least here, our horizon test package just doesn't depend on selenium.
> > 
> > This could help: https://review.openstack.org/#/c/35649/
> 
> Could you explain why Selenium is considered to be non-free?

Assuming you're referring to the 'python-selenum' package in Debian,
Google throws up this:

  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=636677

  "the package ships some files which are not yet built from source."

Whether this is still accurate or not, is another matter, since that
bz is 2 years old...

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][Neutron] Mellanox VIF Driver

[Daniel P. Berrange]

On Thu, Jul 04, 2013 at 11:53:45AM +0300, Itzik Brown wrote:
> Hi,
> 
> We released a code for a VIF Driver that will work with Mellanox
> Quantum Plugin.
> Mellanox plugin provisions Virtual Networks via embedded HCA
> switching functionality (SR-IOV capability of Network adapter).
> 
> The code is here:
> https://review.openstack.org/#/c/35189/
> 
> We allocate and assign probed SR-IOV VF (Virtual Function) as a Para
> virtualized Interface of the instance.
> 
> We need to:
> 1)Allocate  device of probed VF
> 2)Give a name to the device in the XML that will allow us to supprot
> live migration.
> 
> We have utility for managing embedded switch that allows VF
> assignment and configuration.
> We chose to use the vif['dev_name'] as a basis for the name of the
> device. This allow us to use different devices on different hosts
> when doing a migration.
> When doing plug/unplug we are using a utility to allocate the device
> and change it's name to the logical one.

My concern / question on the review is about where the allocation
of the device name is done.

Currently you have it done on the Nova side, by calling out the
the utility tools. This ties the Nova VIF code to the specific
impl of the Mellanox driver in Neutron. This means it is not
appropriate to use a generic vif_type=direct, but really need
to ue a vif_type=mellanox.

The other option, would be if you could do the device name
allocation on the Neutron side, in response to the Nova API
call to allocate a VIF port on the network. If this is possible,
then all the Mellanox specific code would be in Neutron, and
the Nova code would be entirely generic, meaning that use of
a generic vif_type=direct would be acceptable.

I favour the latter architectural split, since it de-couples
Nova & Neutron to a greater extent. I'm not sure if is actually
possible to implement it that way, since I'm not familiar with
your Neuton plugin design. If it isn't possible, then you'll
need to change your patch to use a vif_type=mellanox name
instead

> This is basically it.
> It's a temporary solution until there will be a solution that will
> address all the issues regarding SR-IOV.
> Boris Pavlovic is doing work regarding SR-IOV and we plan to
> help/adopt this effort.
> We want to ask for your opinion regarding the way we chose.


Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Change I30b127d6] Cheetah vs Jinja

[Daniel P. Berrange]

On Tue, Jul 16, 2013 at 09:41:55AM -0400, Solly Ross wrote:
> (This email is with regards to https://review.openstack.org/#/c/36316/)
> 
> Hello All,
> 
> I have been implementing the Guru Meditation Report blueprint
> (https://blueprints.launchpad.net/oslo/+spec/guru-meditation-report),
> and the question of a templating engine was raised.  Currently, my
> version of the code includes the Jinja2 templating engine
> (http://jinja.pocoo.org/), which is modeled after the Django
> templating engine (it was designed to be an implementation of the
> Django templating engine without requiring the use of Django), which
> is used in Horizon.  Apparently, the Cheetah templating engine
> (http://www.cheetahtemplate.org/) is used in a couple places in Nova.
> 
> IMO, the Jinja template language produces much more readable templates,
> and I think is the better choice for inclusion in the Report framework.
>  It also shares a common format with Django (making it slightly easier
> to write for people coming from that area), and is also similar to
> template engines for other languages. What does everyone else think?

Repeating my comments from the review...

I don't have an opinion on whether Jinja or Cheetah is a better
choice, since I've essentially never used either of them (beyond
deleting usage of ceetah from libvirt). I do, however, feel we
should not needlessly use multiple different templating libraries
across OpenStack. We should take care to standardize on one option
that is suitable for all our needs. So if the consensus is that
Jinja is better, then IMHO, there would need to be an blueprint
+ expected timeframe to port existing Ceetah usage to use Jinja.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] New DB column or new DB table?

[Daniel P. Berrange]

On Thu, Jul 18, 2013 at 07:05:10AM -0400, Sean Dague wrote:
> On 07/17/2013 10:54 PM, Lu, Lianhao wrote:
> >Hi fellows,
> >
> >Currently we're implementing the BP 
> >https://blueprints.launchpad.net/nova/+spec/utilization-aware-scheduling. 
> >The main idea is to have an extensible plugin framework on nova-compute 
> >where every plugin can get different metrics(e.g. CPU utilization, memory 
> >cache utilization, network bandwidth, etc.) to store into the DB, and the 
> >nova-scheduler will use that data from DB for scheduling decision.
> >
> >Currently we adds a new table to store all the metric data and have 
> >nova-scheduler join loads the new table with the compute_nodes table to get 
> >all the data(https://review.openstack.org/35759). Someone is concerning 
> >about the performance penalty of the join load operation when there are many 
> >metrics data stored in the DB for every single compute node. Don suggested 
> >adding a new column in the current compute_nodes table in DB, and put all 
> >metric data into a dictionary key/value format and store the json encoded 
> >string of the dictionary into that new column in DB.
> >
> >I'm just wondering which way has less performance impact, join load with a 
> >new table with quite a lot of rows, or json encode/decode a dictionary with 
> >a lot of key/value pairs?
> >
> >Thanks,
> >-Lianhao
> 
> I'm really confused. Why are we talking about collecting host
> metrics in nova when we've got a whole project to do that in
> ceilometer? I think utilization based scheduling would be a great
> thing, but it really out to be interfacing with ceilometer to get
> that data. Storing it again in nova (or even worse collecting it a
> second time in nova) seems like the wrong direction.
> 
> I think there was an equiv patch series at the end of Grizzly that
> was pushed out for the same reasons.
> 
> If there is a reason ceilometer can't be used in this case, we
> should have that discussion here on the list. Because my initial
> reading of this blueprint and the code patches is that it partially
> duplicates ceilometer function, which we definitely don't want to
> do. Would be happy to be proved wrong on that.

IIUC, Ceilometer is currently a downstream consumer of data from Nova, but
no functionality in Nova is a consumer of data from Ceilometer. This is good
split from a security separation point of view, since the security of Nova
is self-contained in this architecture.

If Nova schedular becomes dependant on data from ceilometer, then now the
security of Nova depends on the security of Ceilometer, expanding the attack
surface. This is not good architecture IMHO.

At the same time, I hear your concerns about the potential for duplication
of stats collection functionality between Nova & Ceilometer. I don't think
we neccessarily need to remove 100% of duplication. IMHO probably the key
thing is for the virt drivers to expose a standard API for exporting the
stats, and make sure that both ceilometer & nova schedular use the same
APIs and ideally the same data feed, so we're not invoking the same APIs
twice to get the same data.


Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Python overhead for rootwrap

[Daniel P. Berrange]

On Fri, Aug 02, 2013 at 10:58:11AM +0100, Mark McLoughlin wrote:
> On Thu, 2013-07-25 at 14:40 -0600, Mike Wilson wrote:
> > In my opinion:
> > 
> > 1. Stop using rootwrap completely and get strong argument checking support
> > into sudo (regex).
> > 2. Some sort of long lived rootwrap process, either forked by the service
> > that want's to shell out or a general purpose rootwrapd type thing.
> > 
> > I prefer #1 because it's surprising that sudo doesn't do this type of thing
> > already. It _must_ be something that everyone wants. But #2 may be quicker
> > and easier to implement, my $.02.
> 
> IMHO, #1 set the discussion off in a poor direction.
> 
> Who exactly is stepping up to do this work in sudo? Unless there's
> someone with a even prototype patch in hand, any insistence that we base
> our solution on this hypothetical feature is an unhelpful diversion.
> 
> And even if this work was done, it will be a long time before it's in
> all the distros we support, so improving rootwrap or finding an
> alternate solution will still be an important discussion.

Personally I'm of the opinion that from an architectural POV, use of
either rootwrap or sudo is a bad solution, so arguing about which is
better is really missing the bigger picture. In Linux, there has been
a move away from use of sudo or similar approaches, towards the idea
of having privileged separated services. So if you wanted todo stuff
related to storage, you'd have some small daemon running privilegd,
which exposed APIs over DBus, which the non-privileged thing would
call to make storage changes. Operations exposed by the service would
have access control configured via something like PolicyKit, and/or
SELinux/AppArmour.

Of course this is alot more work than just hacking up some scripts
using sudo or rootwrap. That's the price you pay for properly
engineering formal APIs todo jobs instead of punting to random
shell scripts.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Python overhead for rootwrap

[Daniel P. Berrange]

On Fri, Aug 02, 2013 at 12:50:08PM +0100, Chris Jones wrote:
> Hi
> 
> On 2 August 2013 11:15, Daniel P. Berrange  wrote:
> 
> > better is really missing the bigger picture. In Linux, there has been
> > a move away from use of sudo or similar approaches, towards the idea
> > of having privileged separated services. So if you wanted todo stuff
> >
> 
> I think it would be fair to say that this move has happened significantly
> more in the desktop world than the server world?

Alot of the work has been done by people working to satisfy problems
on the desktop space, eg so you don't have to run admin tools as
root from your desktop. The work they done is useful to the development
to server apps too though. In the virtualization world, you already
have this separation present in any mgmt application (like Nova) which
talks to libvirt for the various privileged operations that are needed
for managing VMs. Nova isn't using as much as it could do though. Nova
isn't using any of libvirt's storage or network related APIs currently,
which could obsolete some of its uses of rootwrap.

> > related to storage, you'd have some small daemon running privilegd,
> > which exposed APIs over DBus, which the non-privileged thing would
> >
> 
> There are several things that worry me about this suggestion:
> 
>  * DBus isn't super pleasing to work with as a developer or a sysadmin

No worse than OpenStack's own RPC system

>  * AIUI it doesn't offer very many guarantees about message delivery or
> high availability

As a host-local service only, I'm not sure high availability is really
a relevant issue.

> > Of course this is alot more work than just hacking up some scripts
> > using sudo or rootwrap. That's the price you pay for properly
> > engineering formal APIs todo jobs instead of punting to random
> > shell scripts.
> >
> 
> Given the sorts of things that OpenStack components need to run with
> privileges, I strongly suspect that even if you wedge DBus in the middle of
> things, you'll still be "punting to random shell scripts" on the backend,
> unless the tools on Linux servers are about to grow a heck of a lot of
> APIs. At that point I'm not really sure you've gained anything other than
> making the whole process more complicated and significantly harder to debug.

There's certainly alot of stuff in Linux mgmt which is lacking any kind
of reasonable API. We have felt that pain in providing APIs in libvirt,
but I still think it is better to have your root/non-root barrier defined
in terms of APIs. It is much simpler to validate well defined parameters
to API calls, than to parse & validate shell command lines. Shell commands
have a nasty habit of changing over time, or being inconsistent across
distros, or have ill defined error handling / reporting behaviour.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Blueprint for Nova native image building

[Daniel P. Berrange]

On Tue, Aug 06, 2013 at 12:20:00PM -0400, Russell Bryant wrote:
> On 08/06/2013 11:53 AM, Ian Mcleod wrote:
> > Hello,
> > 
> > A blueprint has been registered regarding API additions to Nova to
> > enable the creation of base images from external OS install sources.
> > This provides a way to build images from scratch via native OS installer
> > tools using only the resources provided through Nova.  These images can
> > then be further customized by other tools that expect an existing image
> > as an input, such as disk image builder.
> > 
> > Blueprint -
> > https://blueprints.launchpad.net/nova/+spec/base-image-creation
> > 
> > Specification - https://wiki.openstack.org/wiki/NovaImageCreationAPI
> > 
> > If this is a topic that interests you, please have a look (the spec is
> > not very long) and join the conversation.
> > 
> > Please note that this blueprint follows on from proof of concept work
> > for native image building discussed on this list in April:
> > 
> > http://lists.openstack.org/pipermail/openstack-dev/2013-April/007157.html
> 
> Thanks of the update on this work.
> 
> I see that your proof of concept shows how this can work as a tool
> outside of Nova:
> 
> https://github.com/redhat-openstack/image-building-poc
> 
> So, my biggest question is whether or not it makes sense for this to be
> a Nova feature or not.  If something can be implemented as a consumer of
> Nova, my default answer is that it should stay outside of nova until I
> am convinced otherwise.  :-)
> 
> It sounds like this is mostly an extension to nova that implements a
> series of operations that can be done just as well outside of Nova.  Are
> there enhancements you are making or scenarios that won't work at all
> unless it lives inside of Nova?
> 
> If it doesn't end up on the server side, it could potentially be
> implemented as an extension to novaclient.

I think the key thing is that want to make sure we don't have all
clients apps having to re-invent the wheel. The way the proof of
concept was done as a standalone tool, would entail such wheel
re-invention by any frontend to Nova like the 'nova' cli and
Horizon dashboard. Possibly you could mitigate that if you could
actually put all the smarts in the python-novaclient library API
so it was shared by all frontend apps.

IIUC, though there is some state information that it is desirable
to maintain while the images are being built. You'd probably
such state visible to all clients talking to the same nova instance,
not hidden away in the client side where its only visible to that
single frontend instance.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Blueprint for Nova native image building

[Daniel P. Berrange]

On Wed, Aug 07, 2013 at 10:34:57AM -0400, Russell Bryant wrote:
> On 08/06/2013 06:06 PM, Ian McLeod wrote:
> > On Tue, 2013-08-06 at 16:02 -0300, Monty Taylor wrote:
> > The proof of concept approach is limited to full-virt hypervisors.  It's
> > unclear to me if there's a way we can make this work for Xen-backed
> > installs without the kind of lower level access to the virt environment
> > that we'll get if we exist inside of Nova.
> 
> Can you write up some more detail on this point?
> 
> > More generally, it's likely that we'll have more flexibility to behave
> > in a sane/optimized manner based on backing hypervisor if the code is
> > inside of Nova.  For example, we've talked about improving our detection
> > of a failed install by monitoring disk IO.
> 
> If we were to service-ify this, it would be interesting to look into
> using Ceilometer for monitoring.

NB, 

> 
> 
>  It sounds like this is mostly an extension to nova that implements a
>  series of operations that can be done just as well outside of Nova.  Are
>  there enhancements you are making or scenarios that won't work at all
>  unless it lives inside of Nova?
> > 
> > Other than the Xen issue above, I'm not sure there's anything that
> > simply won't work at all, though there are things that perhaps won't
> > scale as well or won't run as quickly.
> 
> Why would it be slower?

I don't think there's any particular reason why Xen should be slower
or less scalable from an architectural POV here. Any perf differences
would be just those inherant to the hypervisor platform in question.

> How about scale?  Just the increased API load?  I wouldn't expect this
> to be something done frequently.  It's more API calls, but removes a
> long running task from inside nova (longer than anything else that
> exists in nova today).

In terms of load, all the heavy I/O & CPU burn would be in the context
of a VM running in nova. So I don't think this approach to image building
is would be introducing any new architectural scalability problems. Indeed
this is the main attraction of running the OS installer inside a VM managed
by Nova - the image builder just takes advantage of all Nova's support for
resource manager/VM scheduler placement etc.

> >> Yes to everything Russel said. I'd like to see the tool be standalone.
> >> Then, if there is a desire to provide the ability to run it via an api,
> >> the tool could be consumed (similar discussions have happened around
> >> putting diskimage-builder behind a service as well)
> >>
> >> That said - if we did service-ify the tool, wouldn't glance be a more
> >> appropriate place for that sort of thing?
> > 
> > Possibly, though the proof of concept (and we hope our proposed
> > nova-based re-implementation) can build both glance images and cinder
> > volume backed images.
> 
> I like this idea (glance seeming to make more sense conceptually).

I've gone back & forth with thinking about whether it makes sense in
glance or nova, and don't have a strong opinion either way really.
>From a technical POV I think it could be made to work in either without
much bother.

> It seems like the main sticking point is whether or not it can be made
> to work for all (or most) hypervisors from outside of nova.  Can we dig
> into this point a bit deeper?

I think that it ought to be possible to make it work for any hypervisor
that is doing full-machine virt (ie not container drivers like LXC). We
may not have sufficient APIs, or we may not have enough features implemented
in some virt drivers, but that's just a case of donkey work, rather than
any architectural blocker.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Blueprint for Nova native image building

[Daniel P. Berrange]

On Wed, Aug 07, 2013 at 10:44:29AM -0400, Russell Bryant wrote:
> On 08/07/2013 10:34 AM, Russell Bryant wrote:
> >>> That said - if we did service-ify the tool, wouldn't glance be a more
> >>> appropriate place for that sort of thing?
> >>
> >> Possibly, though the proof of concept (and we hope our proposed
> >> nova-based re-implementation) can build both glance images and cinder
> >> volume backed images.
> > 
> > I like this idea (glance seeming to make more sense conceptually).
> > 
> > It seems like the main sticking point is whether or not it can be made
> > to work for all (or most) hypervisors from outside of nova.  Can we dig
> > into this point a bit deeper?
> 
> I forgot to mention here that I think targeting glance only as the
> destination is fine.  You can create cinder volumes from glance images
> easily.

That makes sense; only producing glance images as output, avoids needlessly
re-inventing functionality already available elsewhere in our APIs.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

[Daniel P. Berrange]

On Wed, Aug 07, 2013 at 05:43:19PM +, Joshua Harlow wrote:
> Interesting, I should look into conary.
> 
> I was hoping that DNF (the yum rewrite) would be a little better at this
> stage, but it still seems a ways off.

I don't know about what future plans the developers may have, but
currently DNF is just offering the same functionality as Yum, but
with its depsolver using a C library to make it orders of magnitude
faster. It isn't attempting to manage packages in any different
way to what Yum already does

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

[Daniel P. Berrange]

On Tue, Aug 06, 2013 at 11:36:44PM -0300, Monty Taylor wrote:
> 
> 
> On 08/06/2013 11:14 PM, Robert Collins wrote:
> > On 7 August 2013 11:22, Jay Buffington  wrote:
> > 
> >> ln -s /usr/lib64/python2.6/site-packages/libvirtmod_qemu.so
> >> $(VENV)/lib/python2.6/site-packages/
> >>
> >> Why isn't libvirt-python on pypi?  AFAICT, nothing is stopping us from
> >> uploading it.  Maybe we should just stick it on there and this issue
> >> will be resolved once and for all.
> > 
> > Please please oh yes please :).
> 
> It doesn't build from a setup.py, so there is nothing to upload. It's
> built as part of the libvirt C library, and its build depends on scripts
> that autogenerate source code from the C library headers (think swig,
> except homegrown)

Yep, 90% of the libvirt python module is auto-generated based on
artifacts from the C library build. That doesn't mean to say it
could not be separated out from the main package, but it not going
to be a quick or simple job. Anyone wanting to volunteer to do this
kind of thing needs to start a discussion on the libvirt mailing
list with their proposals to get buy in from upstream libvirt
maintainers.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Osl and dangerous code merging

[Daniel P. Berrange]

On Thu, Aug 08, 2013 at 11:39:44AM +0100, Mark McLoughlin wrote:
> What do you mean by "dangerous code merging" in the subject? The body of
> your mail doesn't make any reference to whatever "danger" you're seeing.
> 
> On Thu, 2013-08-08 at 14:16 +0400, Boris Pavlovic wrote:
> > Hi All,
> > 
> > Could somebody answer me, why we are merging oslo code in other projects
> > and don't use
> > git submodules (http://git-scm.com/book/en/Git-Tools-Submodules)
> 
> The idea of using submodules has come a few times. I don't have a
> fundamental objection to it, except any time I've seen submodules used
> in a project they've been extremely painful for everyone involved.
> 
> I'd be happy to look at a demo of a submodule based system for projects
> to use code from oslo-incubator.

submodules certainly could work as a way to avoid the cut+paste
approach we currently do. I agree though that they do add an
extra layer of pain & suffering for developers who don't fully
understand what they're doing. We use them in libvirt and try
to hide the pain behind clever scripts which attempt to keep
the submodule properly synced, but we still get pretty frequent
problem reports from devs who've managed to get themselves into
a mess with the submodule state.

If we want to improve our interaction with oslo then IMHO more
effort should be spent on turning bits of oslo-incubator into
stable, standalone modules, removing the cut+paste need entirely.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Blueprint for Nova native image building

[Daniel P. Berrange]

On Thu, Aug 08, 2013 at 09:28:51AM -0500, Ian McLeod wrote:
> On Wed, 2013-08-07 at 12:05 -0400, Russell Bryant wrote:
> > On 08/07/2013 10:46 AM, Daniel P. Berrange wrote:
> > > On Tue, Aug 06, 2013 at 12:20:00PM -0400, Russell Bryant wrote:
> > >> On 08/06/2013 11:53 AM, Ian Mcleod wrote:
> > >>> Hello,
> > >>>
> > >>> A blueprint has been registered regarding API additions to Nova to
> > >>> enable the creation of base images from external OS install sources.
> > >>> This provides a way to build images from scratch via native OS installer
> > >>> tools using only the resources provided through Nova.  These images can
> > >>> then be further customized by other tools that expect an existing image
> > >>> as an input, such as disk image builder.
> > >>>
> > >>> Blueprint -
> > >>> https://blueprints.launchpad.net/nova/+spec/base-image-creation
> > >>>
> > >>> Specification - https://wiki.openstack.org/wiki/NovaImageCreationAPI
> > >>>
> > >>> If this is a topic that interests you, please have a look (the spec is
> > >>> not very long) and join the conversation.
> > >>>
> > >>> Please note that this blueprint follows on from proof of concept work
> > >>> for native image building discussed on this list in April:
> > >>>
> > >>> http://lists.openstack.org/pipermail/openstack-dev/2013-April/007157.html
> > >>
> > >> Thanks of the update on this work.
> > >>
> > >> I see that your proof of concept shows how this can work as a tool
> > >> outside of Nova:
> > >>
> > >> https://github.com/redhat-openstack/image-building-poc
> > >>
> > >> So, my biggest question is whether or not it makes sense for this to be
> > >> a Nova feature or not.  If something can be implemented as a consumer of
> > >> Nova, my default answer is that it should stay outside of nova until I
> > >> am convinced otherwise.  :-)
> > >>
> > >> It sounds like this is mostly an extension to nova that implements a
> > >> series of operations that can be done just as well outside of Nova.  Are
> > >> there enhancements you are making or scenarios that won't work at all
> > >> unless it lives inside of Nova?
> > >>
> > >> If it doesn't end up on the server side, it could potentially be
> > >> implemented as an extension to novaclient.
> > > 
> > > I think the key thing is that want to make sure we don't have all
> > > clients apps having to re-invent the wheel. The way the proof of
> > > concept was done as a standalone tool, would entail such wheel
> > > re-invention by any frontend to Nova like the 'nova' cli and
> > > Horizon dashboard. Possibly you could mitigate that if you could
> > > actually put all the smarts in the python-novaclient library API
> > > so it was shared by all frontend apps.
> > 
> > Yeah, I was thinking python-novaclient.  The 'nova' command line tool is
> > just a wrapper around the library portion.
> > 
> > > IIUC, though there is some state information that it is desirable
> > > to maintain while the images are being built. You'd probably
> > > such state visible to all clients talking to the same nova instance,
> > > not hidden away in the client side where its only visible to that
> > > single frontend instance.
> > 
> > That's an interesting point.  Do we care about having an image build
> > executing by the command line to show up in the UI as an image build?
> > Right now it's just going to be another nova instance.  We have to do it
> > on the server side to do any better.  I'm not even sure it could
> > integrate well with Horizon doing it in python-novaclient.  I don't
> > think you could start another session and see the operation in progress.
> 
> Perhaps it's worth revisiting the basic question:
> 
> Is scratch image building (as distinct from run time customization) a
> task that benefits from being exposed as a distinct activity available
> to a typical Horizon user or as structured resource available to other
> services via REST?
> 
> The blueprint assumes the answer is yes.
> 
> But, is scratch image building in fact low level and infrequent enough
> to live happily as a standalone tool?  (Oz for Nova, if you will.)

I don't think frequency of usage is the right question to evaluate
this really. Even if it is only used by a small set of users,
if those users are using Horizon for their work, IMHO, they will
rightly expect image building to fit into Horizon, not have to go
to a separate standalone tool for the job.


Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Code review study

[Daniel P. Berrange]

On Thu, Aug 15, 2013 at 09:42:09PM +0930, Christopher Yeoh wrote:
> On Thu, Aug 15, 2013 at 11:42 AM, Robert Collins
> wrote:
> 
> > This may interest data-driven types here.
> >
> >
> > https://www.ibm.com/developerworks/rational/library/11-proven-practices-for-peer-review/
> >
> > Note specifically the citation of 200-400 lines as the knee of the review
> > effectiveness curve: that's lower than I thought - I thought 200 was
> > clearly fine - but no.
> >
> >
> Very interesting article. One other point which I think is pretty relevant
> is point 4 about getting authors to annotate the code better (and for those
> who haven't read it, they don't mean comments in the code but separately)
> because it results in the authors picking up more bugs before they even
> submit the code.
> 
> So I wonder if its worth asking people to write more detailed commit logs
> which include some reasoning about why some of the more complex changes
> were done in a certain way and not just what is implemented or fixed. As it
> is many of the commit messages are often very succinct so I think it would
> help on the review efficiency side too.

We in fact already ask people to do that 

  
https://wiki.openstack.org/wiki/GitCommitMessages#Information_in_commit_messages

Commit message quality has improved somewhat since I first wrote & published
that page, but there's definitely still scope to improve things further. What
it really needs is for more reviewers to push back against badly written
commit messages, to nudge authors into the habit of being more verbose in
their commits.

That said, commit messages are fine for describing what a specific commit
does, but for many non-trivial features spanning multiple commits, you need
an external reference document. In projects based around a git send-mail
mailing list review process, there would be cover letter describing the
broad design / goals of the work. In openstack's dev model, this role is
really served by the blueprints.

Unfortunately we are often badly lacking quality control over blueprints.
Far too many blueprints have just a handful of lines of text in the blueprint
description and no link to any wiki page with design info or background on
why certain decisions where made. At some point I think it would be pretty
beneficial to start getting stricter around quality of blueprints info, so
that the creation of the blueprint isn't just a "tick box" for process
compliance, but an actual useful source of information.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Code review study

[Daniel P. Berrange]

On Thu, Aug 15, 2013 at 09:46:07AM -0500, Dolph Mathews wrote:
> On Thu, Aug 15, 2013 at 7:57 AM, Christopher Yeoh  wrote:
> 
> > On Thu, Aug 15, 2013 at 9:54 PM, Daniel P. Berrange 
> > wrote:Commit message quality has improved somewhat 
> > since I first wrote &
> > published
> >
> >  that page, but there's definitely still scope to improve things further.
> >> What
> >> it really needs is for more reviewers to push back against badly written
> >> commit messages, to nudge authors into the habit of being more verbose in
> >> their commits.
> >>
> >>
> > Agreed. There is often "what" and sometimes "why", but not very often
> > "how" in commit messages.
> >
> 
> ++
> 
> Beyond the one line summary (which *should* describe "what" changed),
> describing "what" changed in the commit message is entirely redundant with
> the commit itself.

It isn't that clearcut actually. It is quite often helpful to summarize
"what" changed in the commit message, particularly for changes touching
large areas of code, or many files. The diff's can't always be assumed
to be easily readable - for example if you re-indented a large area of
code, the actual "what" can be clear as mud. Or if there are related
changes spread across many files & functions, a description of what is
being done will aid reviewers. Just be pragmatic about deciding when
a change is complex enough that it merits summarizing the 'what', as
well as the 'why'.


Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] live-snapshot/cloning of virtual machines

[Daniel P. Berrange]

On Wed, Aug 14, 2013 at 04:53:01PM -0700, Vishvananda Ishaya wrote:
> Hi Everyone,
> 
> I have been trying for some time to get the code for the live-snapshot 
> blueprint[1]
> in. Going through the review process for the rpc and interface code[2] was 
> easy. I
> suspect the api-extension code[3] will also be relatively trivial to get in. 
> The
> main concern is with the libvirt driver implementation[4]. I'd like to 
> discuss the
> concerns and see if we can make some progress.
> 
> Short Summary (tl;dr)
> =
> 
> I propose we merge live-cloning as an experimental feature for havanna and 
> have the
> api extension disabled by default.
> 
> Overview
> 
> 
> First of all, let me express the value of live snapshoting. The slowest part 
> of the
> vm provisioning process is generally booting of the OS. The advantage of live-
> snapshotting is that it allows the possibility of bringing up application 
> servers
> while skipping the overhead of vm (and application startup).

For Linux at least I think bootup time is a problem that is being solved by the
distros. It is possible to boot up many modern Linux distros in a couple of 
seconds
even in physical hardware - VMs can be even faster since they don't have such 
stupid
BIOS to worry about & have a restricted set of possible hardware. This is on a 
par
with, or better than, the overheads imposed by Nova itself in the boot up 
process.

Windows may be a different story, but I've not used it in years so don't know 
what
its boot performance is like.

> I recognize that this capability comes with some security concerns, so I 
> don't expect
> this feature to go in and be ready to for use in production right away. 
> Similarly,
> containers have a lot of the same benefit, but have had their own security 
> issues
> which are gradually being resolved. My hope is that getting this feature in 
> would
> allow people to start experimenting with live-booting so that we could 
> uncover some
> of these security issues.
> 
> There are two specific concerns that have been raised regarding my patch. The 
> first
> concern is related to my use of libvirt. The second concern is related to the 
> security
> issues above. Let me address them separately.
> 
> 1. Libvirt Issues
> =
> 
> The only feature I require from the hypervisor is to load memory/processor 
> state for
> a vm from a file. Qemu supports this directly. The only way that libvirt 
> exposes this
> functionality is via its restore command which is specifically for restoring 
> the
> previous state of an existing vm. "Cloning", or restoring the memory state of 
> a
> cloned vm is considered unsafe (which I will address in the second point, 
> below).
> 
> The result of the limited api is that I must include some hacks to make the 
> restore
> command actually allow me to restore the state of the new vm. I recognize 
> that this
> is using an undocumented libvirt api and isn't the ideal solution, but it 
> seemed
> "better" then avoiding libvirt and talking directly to qemu.
> 
> This is obviously not ideal. It is my hope that this 0.1 version of the 
> feature will
> allow us to iteratively improve the live-snapshot/clone proccess and get the 
> security
> to a point where the libvirt maintainers would be willing to accept a patch 
> to directly
> expose an api to load memory from a file.

To characterize this as a libvirt issue is somewhat misleading. The reason why 
libvirt
does not explicitly allow this, is that from discussions with the upstream 
QEMU/KVM
developers, the recommendation/advise that this is not a safe operation and 
should not
be exposed to application developers.

The expectation is that the functionality in QEMU is only targetted for taking 
point in
time snapshots & allowing rollback of a VM to those snapshots, not creating 
clones of
active VMs.

> 2. Security Concerns
> 
> 
> There are a number of security issues with loading state from another vm. 
> Here is a
> short list of things that need to be done just to make a cloned vm usable:
> 
> a) mac address needs to be recreated
> b) entropy pool needs to be reset
> c) host name must be reset
> d) host keys bust be regenerated
> 
> There are others, and trying to clone a running application as well may 
> expose other
> sensitive data, especially if users are snaphsoting vms and making them 
> public.
> 
> The only issue that I address on the driver side is the mac addresses. This 
> is the
> minimum that needs to be done just to be able to access the vm over the 
> network. This
> is implemented by unplugging all network devices before the snapshot and 
> plugging new
> network devices in on clone. This isn't the most friendly thing to guest 
> applications,
> but it seems like the safest option for the first version of this feature.

This is not really as safe as you portray. When restoring from the snapshot the 
VM
will initially be running with virtual NIC with a different

Re: [openstack-dev] [nova] live-snapshot/cloning of virtual machines

[Daniel P. Berrange]

On Mon, Aug 19, 2013 at 08:28:58AM +1200, Robert Collins wrote:
> On 17 August 2013 07:01, Russell Bryant  wrote:
> 
> >> Maybe we've grown up to the point where we have to be more careful and
> >> not introduce
> >> these kind of features and the maintenance cost of introducing
> >> experimental features is
> >> too great. If that is the community consensus, then I'm happy keep the
> >> live snapshot stuff
> >> in a branch on github for people to experiment with.
> >
> > My feeling after following this discussion is that it's probably best to
> > keep baking in another branch (github or whatever).  The biggest reason
> > is because of the last comment quoted from Daniel Berrange above.  I
> > feel that like that is a pretty big deal.
> 
> So, reading between the lines here, I guess you're worried that we'd
> let code paths that violate what upstream will support leak into the
> main codepaths for libvirt - and thus we'd end up with a situation
> where we aren't supported by upstream for all regular operations.

Yes, if you perform a live clone of a VM, then you have effectively
tainted that VM for the rest of its lifetime. From the virt host
vendors' POV, any unexpected or problematic behaviour you get from
that VM thereafter will be outside scope of support. The onus would
be on the openstack sysadmin to demonstrate that the same problem
occurs without the use of live cloning.

Running a production cloud using a feature that your virt host
vendor considers unsupported would be somewhat reckless IMHO, unless
you think your sysadmins think they have the skills to solve all
possible problems in that area themselves, which is unlikely for most
cloud vendors.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] live-snapshot/cloning of virtual machines

[Daniel P. Berrange]

On Mon, Aug 19, 2013 at 01:28:25AM -0700, Tim Smith wrote:
> On Sun, Aug 18, 2013 at 1:28 PM, Robert Collins
> wrote:
> 
> > On 17 August 2013 07:01, Russell Bryant  wrote:
> >
> > >> Maybe we've grown up to the point where we have to be more careful and
> > >> not introduce
> > >> these kind of features and the maintenance cost of introducing
> > >> experimental features is
> > >> too great. If that is the community consensus, then I'm happy keep the
> > >> live snapshot stuff
> > >> in a branch on github for people to experiment with.
> > >
> > > My feeling after following this discussion is that it's probably best to
> > > keep baking in another branch (github or whatever).  The biggest reason
> > > is because of the last comment quoted from Daniel Berrange above.  I
> > > feel that like that is a pretty big deal.
> >
> > So, reading between the lines here, I guess you're worried that we'd
> > let code paths that violate what upstream will support leak into the
> > main codepaths for libvirt - and thus we'd end up with a situation
> > where we aren't supported by upstream for all regular operations.
> >
> > I agree that *that* is a big deal : is there something we could do to
> > prevent that happening? E.g. annotating this whole thing as
> > experimental/not upstream supported or something?
> >
> 
> My understanding is that the live-snapshot extensions are to be disabled by
> default (either on the driver side or the API side) and must be explicitly
> enabled via a nova configuration change. Thus, it is not interferant with
> mainline codepaths by default, and the user will not place themself into an
> "unmaintainable" position unless they or their OpenStack distro provider
> flips that particular switch.
>
> That would seem to me a sufficiently high bar for preventing the user from
> shooting themselves in the foot. As for any objections to the mere
> _existence_ of the live-snapshot capability, I propose that the market
> answer the question as to whether or not the feature has value.

NB live-snapshots are fine if used for rollback/forward of an individual
VMs. It is only use of live snapshots for the purpose of cloning VMs
that's the issue here. Whether the market wants live cloning or not is
irrelevant until a vendor will actually support it from a technology POV

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Stats on blueprint design info / creation times

[Daniel P. Berrange]

In this thread about code review:

  http://lists.openstack.org/pipermail/openstack-dev/2013-August/013701.html

I mentioned that I thought there were too many blueprints created without
sufficient supporting design information and were being used for "tickbox"
process compliance only. I based this assertion on a gut feeling I have
from experiance in reviewing.

To try and get a handle on whether there is truely a problem, I used the
launchpadlib API to extract some data on blueprints [1].

In particular I was interested in seeing:

  - What portion of blueprints have an URL containing an associated
design doc,

  - How long the descriptive text was in typical blueprints

  - Whether a blueprint was created before or after the dev period
started for that major release.


The first two items are easy to get data on. On the second point, I redid
line wrapping on description text to normalize the line count across all
blueprints. This is because many blueprints had all their text on one
giant long line, which would skew results. I thus wrapped all blueprints
at 70 characters.

The blueprint creation date vs release cycle dev start date is a little
harder. I inferred the start date of each release, by using the end date
of the previous release. This is probably a little out but hopefully not
by enough to totally invalidate the usefulness of the stats below. Below,
"Early" means created before start of devel, "Late" means created after
the start of devel period.

The data for the last 3 releases is:

  Series: folsom
Specs: 178
Specs (no URL): 144
Specs (w/ URL): 34
Specs (Early): 38
Specs (Late): 140
Average lines: 5
Average words: 55


  Series: grizzly
Specs: 227
Specs (no URL): 175
Specs (w/ URL): 52
Specs (Early): 42
Specs (Late): 185
Average lines: 5
Average words: 56


  Series: havana
Specs: 415
Specs (no URL): 336
Specs (w/ URL): 79
Specs (Early): 117
Specs (Late): 298
Average lines: 6
Average words: 68


Looking at this data there are 4 key take away points

  - We're creating more blueprints in every release.

  - Less than 1 in 4 blueprints has a link to a design document. 

  - The description text for blueprints is consistently short
(6 lines) across releases.

  - Less than 1 in 4 blueprints is created before the devel
period starts for a release.


You can view the full data set + the script to generate the
data which you can look at to see if I made any logic mistakes:

  http://berrange.fedorapeople.org/openstack-blueprints/


There's only so much you can infer from stats like this, but IMHO think the
stats show that we ought to think about how well we are using blueprints as
design / feature approval / planning tools.


That 3 in 4 blueprint lack any link to a design doc and have only 6 lines of
text description, is a cause for concern IMHO. The blueprints should be giving
code reviewers useful background on the motivation of the dev work & any
design planning that took place. While there are no doubt some simple features
where 6 lines of text is sufficient info in the blueprint, I don't think that
holds true for the majority.

In addition to helping code reviewers, the blueprints are also arguably a
source of info for QA people testing OpenStack and for the docs teams
documenting new features in each release. I'm not convinced that there is
enough info in many of the blueprints to be of use to QA / docs people.


The creation dates of the blueprints are also an interesting data point.
If the design summit is our place for reviewing blueprints and 3 in 4
blueprints in a release are created after the summit, that's alot of
blueprints potentially missing summit discussions. On the other hand many
blueprints will have corresponding discussions on mailing lists too,
which is arguably just as good, or even better than, summit discussions.

Based on the creation dates though & terseness of design info, I think
there is a valid concern here that blueprints are being created just for
reason of "tickbox" process compliance. 

In theory we have an approval process for blueprints, but are we ever
rejecting code submissions for blueprints which are not yet approved ?
I've only noticed that happen a couple of times in Nova for things that
were pretty clearly controversial.

I don't intend to suggest that we have strict rules that all blueprints
must be min X lines of text, or be created by date Y. It is important
to keep the flexibility there to avoid development being drowned in
process without benefits.

I do think we have scope for being more rigourous in our review of
blueprints, asking people to expand on the design info associated with
a blueprint. Perhaps also require that a blueprint is actually approved
by the core team before we go to the trouble of reviewing & approving
the code implementing a blueprint in Gerrit.

Regards,
Daniel

[1] http://berrange.fedorapeople.org/openstack-blueprints/bluepri

Re: [openstack-dev] Code review study

[Daniel P. Berrange]

On Tue, Aug 20, 2013 at 04:02:12PM +0100, Mark McLoughlin wrote:
> On Tue, 2013-08-20 at 11:26 +0100, Mark McLoughlin wrote:
> > On Thu, 2013-08-15 at 14:12 +1200, Robert Collins wrote:
> > > This may interest data-driven types here.
> > > 
> > > https://www.ibm.com/developerworks/rational/library/11-proven-practices-for-peer-review/
> > > 
> > > Note specifically the citation of 200-400 lines as the knee of the review
> > > effectiveness curve: that's lower than I thought - I thought 200 was
> > > clearly fine - but no.
> > 
> > The full study is here:
> > 
> > http://support.smartbear.com/resources/cc/book/code-review-cisco-case-study.pdf
> > 
> > This is an important subject and I'm glad folks are studying it, but I'm
> > sceptical about whether the "Defect density vs LOC" is going to help us
> > come up with better guidelines than we have already.
> > 
> > Obviously, a metric like LOC hides some serious subtleties. Not all
> > changes are of equal complexity. We see massive refactoring patches
> > (like s/assertEquals/assertEqual/) that are actually safer than gnarly,
> > single-line, head-scratcher bug-fixes. The only way the report addresses
> > that issue with the underlying data is by eliding >10k LOC patches.
> > 
> > The "one logical change per commit" is a more effective guideline than
> > any LOC based guideline:
> > 
> > https://wiki.openstack.org/wiki/GitCommitMessages#Structural_split_of_changes
> > 
> > IMHO, the number of distinct logical changes in a patch has a more
> > predictable impact on review effectiveness than the LOC metric.
> 
> Wow, I didn't notice Joe had started to enforce that here:
> 
>   https://review.openstack.org/41695
> 
> and the exact example I mentioned above :)
> 
> We should not enforce rules like this blindly.

Agreed, lines of code is a particularly poor metric for evaluating
commits on. 


Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Stats on blueprint design info / creation times

[Daniel P. Berrange]

On Tue, Aug 20, 2013 at 10:36:39AM -0500, Anne Gentle wrote:
> On Mon, Aug 19, 2013 at 10:47 AM, Daniel P. Berrange 
> wrote:
> > The data for the last 3 releases is:
> >
> >   Series: folsom
> > Specs: 178
> > Specs (no URL): 144
> > Specs (w/ URL): 34
> > Specs (Early): 38
> > Specs (Late): 140
> > Average lines: 5
> > Average words: 55
> >
> >
> >   Series: grizzly
> > Specs: 227
> > Specs (no URL): 175
> > Specs (w/ URL): 52
> > Specs (Early): 42
> > Specs (Late): 185
> > Average lines: 5
> > Average words: 56
> >
> >
> >   Series: havana
> > Specs: 415
> > Specs (no URL): 336
> > Specs (w/ URL): 79
> > Specs (Early): 117
> > Specs (Late): 298
> > Average lines: 6
> > Average words: 68
> >
> >
> > Looking at this data there are 4 key take away points
> >
> >   - We're creating more blueprints in every release.
> >
> >   - Less than 1 in 4 blueprints has a link to a design document.
> >
> >   - The description text for blueprints is consistently short
> > (6 lines) across releases.
> >
> >
> Thanks for running the numbers. My instinct told me this was the case, but
> the data is especially helpful. Sometimes six lines is enough, but mostly I
> rely on the linked spec for writing docs. If those links are at 25% that's
> a bad trend.
> 
> 
> >   - Less than 1 in 4 blueprints is created before the devel
> > period starts for a release.
> >
> >
> I find this date mismatch especially intriguing, because the Foundation and
> member company sponsors spend millions on Design Summits annually and
> caters so much to getting people together in person. Yet the blueprints
> aren't created in enough detail for discussion before the Summit dates? Is
> that really what the data says? Is any one project skewing this (as in,
> they haven't been at a Summit or they don't follow integrated release
> dates?)
> 
> I'll dig in further to the data set below.
> 
> 
> >
> > You can view the full data set + the script to generate the
> > data which you can look at to see if I made any logic mistakes:
> >
> >   http://berrange.fedorapeople.org/openstack-blueprints/
> >
> >
> >
> I wouldn't think to include marconi in the dataset as they've just asked
> about incubation in June 2013. I think you excluded keystone. You also want
> ceilometer and oslo if you already included heat. Is it fairly easy to
> re-run? I'd like to see it re-run with the correct program listings.
> 
> Also please rerun with Swift excluded as their release dates are not on the
> mark with the other projects. I'd like more info around the actual timing.

Ok, I've changed the projects it analyses per your recommendation.
Also I've made it print the cut off date I used to assign blueprints
to the "early" or "late" creation date buckets

The overall results are approximately the same though

Series: folsom all
  Specs: 177
  Specs (no URL): 145
  Specs (w/ URL): 32
  Specs (Before 2012-04-05 14:43:29.870782+00:00): 39
  Specs (After 2012-04-05 14:43:29.870782+00:00): 138
  Average lines: 5
  Average words: 54


Series: grizzly all
  Specs: 255
  Specs (no URL): 187
  Specs (w/ URL): 68
  Specs (Before 2012-09-27 00:00:00+00:00): 47
  Specs (After 2012-09-27 00:00:00+00:00): 208
  Average lines: 6
  Average words: 61


Series: havana all
  Specs: 470
  Specs (no URL): 379
  Specs (w/ URL): 91
  Specs (Before 2013-04-04 12:59:00+00:00): 137
  Specs (After 2013-04-04 12:59:00+00:00): 333
  Average lines: 6
  Average words: 69


I also produced summary stats per project. Showing Nova 


Series: folsom nova
  Specs: 54
  Specs (no URL): 37
  Specs (w/ URL): 17
  Specs (Before 2012-04-05 14:43:29.870782+00:00): 20
  Specs (After 2012-04-05 14:43:29.870782+00:00): 34
  Average lines: 6
  Average words: 61


Series: grizzly nova
  Specs: 68
  Specs (no URL): 51
  Specs (w/ URL): 17
  Specs (Before 2012-09-27 00:00:00+00:00): 17
  Specs (After 2012-09-27 00:00:00+00:00): 51
  Average lines: 6
  Average words: 65


Series: havana nova
  Specs: 131
  Specs (no URL): 107
  Specs (w/ URL): 24
  Specs (Before 2013-04-04 12:59:00+00:00): 31
  Specs (After 2013-04-04 12:59:00+00:00): 100
  Average lines: 7
  Average words: 72


And keystone



Series: folsom keystone
  Specs: 9
  Specs (no URL): 8
  Specs (w/ URL): 1
  Specs (Before 2012-04-05 14:43:29.870782+00:00): 3
  Specs (After 2012-04-05 14:43:29.870782+00:00): 6
  Average lines: 4
  Average words: 37


Series: grizzly keystone
  Specs: 16
  Specs (no URL)

Re: [openstack-dev] Stats on blueprint design info / creation times

[Daniel P. Berrange]

On Tue, Aug 20, 2013 at 12:53:25PM -0300, Thierry Carrez wrote:
> Anne Gentle wrote:
> >   - Less than 1 in 4 blueprints is created before the devel
> > period starts for a release.
> > 
> > I find this date mismatch especially intriguing, because the Foundation
> > and member company sponsors spend millions on Design Summits annually
> > and caters so much to getting people together in person. Yet the
> > blueprints aren't created in enough detail for discussion before the
> > Summit dates? Is that really what the data says? Is any one project
> > skewing this (as in, they haven't been at a Summit or they don't follow
> > integrated release dates?)
> 
> That does not surprise me. A lot of people do not link a blueprint to
> their session proposal on the design summit session suggestion system --
> sometimes it's the discussion itself which allows to formulate the right
> blueprints, and those are filed in the weeks just after the summit. And
> I think that's fine.
> 
> It would be more interesting to check how many blueprints are created
> more than two weeks after the design summit. Those would be the late
> blueprints (or the ones created as a tickbox), which escape the release
> planning process.

I'll look up the historic dates for each summit, and try to generate
some stats based on blueprint creation date vs  summit date + 2 weeks.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Stats on blueprint design info / creation times

[Daniel P. Berrange]

On Tue, Aug 20, 2013 at 05:18:21PM +0100, Daniel P. Berrange wrote:
> On Tue, Aug 20, 2013 at 12:53:25PM -0300, Thierry Carrez wrote:
> > Anne Gentle wrote:
> > >   - Less than 1 in 4 blueprints is created before the devel
> > > period starts for a release.
> > > 
> > > I find this date mismatch especially intriguing, because the Foundation
> > > and member company sponsors spend millions on Design Summits annually
> > > and caters so much to getting people together in person. Yet the
> > > blueprints aren't created in enough detail for discussion before the
> > > Summit dates? Is that really what the data says? Is any one project
> > > skewing this (as in, they haven't been at a Summit or they don't follow
> > > integrated release dates?)
> > 
> > That does not surprise me. A lot of people do not link a blueprint to
> > their session proposal on the design summit session suggestion system --
> > sometimes it's the discussion itself which allows to formulate the right
> > blueprints, and those are filed in the weeks just after the summit. And
> > I think that's fine.
> > 
> > It would be more interesting to check how many blueprints are created
> > more than two weeks after the design summit. Those would be the late
> > blueprints (or the ones created as a tickbox), which escape the release
> > planning process.
> 
> I'll look up the historic dates for each summit, and try to generate
> some stats based on blueprint creation date vs  summit date + 2 weeks.

Re-running using the summit date + 2 weeks shift things a little bit.
Here is the  summary for 3 most recent series:


Series: folsom all
  Specs: 177
  Specs (no URL): 145
  Specs (w/ URL): 32
  Specs (Before Mon, 30 Apr 2012 00:00:00 +): 62
  Specs (After Mon, 30 Apr 2012 00:00:00 +): 115
  Average lines: 5
  Average words: 54


Series: grizzly all
  Specs: 255
  Specs (no URL): 187
  Specs (w/ URL): 68
  Specs (Before Sun, 28 Oct 2012 23:00:00 +): 81
  Specs (After Sun, 28 Oct 2012 23:00:00 +): 174
  Average lines: 6
  Average words: 61


Series: havana all
  Specs: 470
  Specs (no URL): 378
  Specs (w/ URL): 92
  Specs (Before Mon, 29 Apr 2013 00:00:00 +): 197
  Specs (After Mon, 29 Apr 2013 00:00:00 +): 273
  Average lines: 6
  Average words: 69


Full data set for version 3 of the stats is now here

  http://berrange.fedorapeople.org/openstack-blueprints/v3/

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Stats on blueprint design info / creation times

[Daniel P. Berrange]

On Wed, Aug 21, 2013 at 11:20:08AM -0400, Doug Hellmann wrote:
> On Mon, Aug 19, 2013 at 11:47 AM, Daniel P. Berrange 
> wrote:
> 
> > In this thread about code review:
> >
> >
> > http://lists.openstack.org/pipermail/openstack-dev/2013-August/013701.html
> >
> > I mentioned that I thought there were too many blueprints created without
> > sufficient supporting design information and were being used for "tickbox"
> > process compliance only. I based this assertion on a gut feeling I have
> > from experiance in reviewing.
> >
> > To try and get a handle on whether there is truely a problem, I used the
> > launchpadlib API to extract some data on blueprints [1].
> >
> > In particular I was interested in seeing:
> >
> >   - What portion of blueprints have an URL containing an associated
> > design doc,
> >
> >   - How long the descriptive text was in typical blueprints
> >
> >   - Whether a blueprint was created before or after the dev period
> > started for that major release.
> >
> >
> > The first two items are easy to get data on. On the second point, I redid
> > line wrapping on description text to normalize the line count across all
> > blueprints. This is because many blueprints had all their text on one
> > giant long line, which would skew results. I thus wrapped all blueprints
> > at 70 characters.
> >
> > The blueprint creation date vs release cycle dev start date is a little
> > harder. I inferred the start date of each release, by using the end date
> > of the previous release. This is probably a little out but hopefully not
> > by enough to totally invalidate the usefulness of the stats below. Below,
> > "Early" means created before start of devel, "Late" means created after
> > the start of devel period.
> >
> > The data for the last 3 releases is:
> >
> >   Series: folsom
> > Specs: 178
> > Specs (no URL): 144
> > Specs (w/ URL): 34
> > Specs (Early): 38
> > Specs (Late): 140
> > Average lines: 5
> > Average words: 55
> >
> >
> >   Series: grizzly
> > Specs: 227
> > Specs (no URL): 175
> > Specs (w/ URL): 52
> > Specs (Early): 42
> > Specs (Late): 185
> > Average lines: 5
> > Average words: 56
> >
> >
> >   Series: havana
> > Specs: 415
> > Specs (no URL): 336
> > Specs (w/ URL): 79
> > Specs (Early): 117
> > Specs (Late): 298
> > Average lines: 6
> > Average words: 68
> >
> >
> > Looking at this data there are 4 key take away points
> >
> >   - We're creating more blueprints in every release.
> >
> >   - Less than 1 in 4 blueprints has a link to a design document.
> >
> >   - The description text for blueprints is consistently short
> > (6 lines) across releases.
> >
> >   - Less than 1 in 4 blueprints is created before the devel
> > period starts for a release.
> >
> >
> > You can view the full data set + the script to generate the
> > data which you can look at to see if I made any logic mistakes:
> >
> >   http://berrange.fedorapeople.org/openstack-blueprints/
> >
> >
> > There's only so much you can infer from stats like this, but IMHO think the
> > stats show that we ought to think about how well we are using blueprints as
> > design / feature approval / planning tools.
> >
> >
> > That 3 in 4 blueprint lack any link to a design doc and have only 6 lines
> > of
> > text description, is a cause for concern IMHO. The blueprints should be
> > giving
> > code reviewers useful background on the motivation of the dev work & any
> > design planning that took place. While there are no doubt some simple
> > features
> > where 6 lines of text is sufficient info in the blueprint, I don't think
> > that
> > holds true for the majority.
> >
> 
> How many of those blueprints without links or expansive documentation are
> related to some other blueprint that does have documentation? In
> ceilometer, we have several blueprint "clusters" where one main blueprint
> has some documentation and the others are present for assigning and
> scheduling the work of a multi-part feature, or vice versa. For example,
> https://blueprints.launchpad.net/ceilometer/+spec/alarming has no real doc
> because it's an "umbrella" blueprint for a lot of other pieces, many of
> which *do* have documentation.

I don't know about ceilometer but for Nova I don't think there are
such a large number of linked blueprints, as to make any significant
difference to the stats here.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Interested in a mid-Icehouse-cycle Nova meet-up?

[Daniel P. Berrange]

On Mon, Aug 19, 2013 at 05:42:00PM -0400, Russell Bryant wrote:
> Greetings,
> 
> Some OpenStack programs have started a nice trend of getting together in
> the middle of the development cycle.  These meetups can serve a number
> of useful purposes: community building, ramping up new contributors,
> tackling hard problems by getting together in the same room, and more.
> 
> I am in the early stages of attempting to plan a Nova meet-up for the
> middle of the Icehouse cycle.  To start, I need to get a rough idea of
> how much interest there is.
> 
> I have very little detail at this point, other than I'm looking at
> locations in the US, and that it would be mid-cycle (January/February).

Is openstack looking to have a strong presence at FOSDEM 2014 ? I didn't
make it to FOSDEM this year, but IIUC, there were quite a few openstack
contributors & talks in 2013.

IOW, should we consider holding the meetup in Brussels just before/after
FOSDEM, so that people who want/need to attend both can try to maximise
utilization of their often limited travel budgets and/or minimise the
number of days lost to travelling ?

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Frustrations with review wait times (was: Requesting feedback on review 35759)

[Daniel P. Berrange]

On Tue, Aug 27, 2013 at 10:31:32AM -0400, Russell Bryant wrote:
> In general, how do we improve these stats?
> 
> 1) Make sure we have a dedicated review team.
> 
> This is something I spend time on regularly.  I try to encourage those
> on the team to regularly participate in reviews.  For those that aren't,
> they generally get removed from the team.  I'm also always on the
> lookout for people we should be adding.
> 
> https://wiki.openstack.org/wiki/Nova/CoreTeam
> 
> 2) Review prioritization
> 
> Reviewers choose which reviews to go after based on all kinds of things
> and some get more or quicker attention than others.  That will always be
> the case.  If there's a lot more interest in X than Y, then X deserves
> more attention, honestly.
> 
> However, many of us could still do a better job prioritizing reviews.
> You can find some tips on that here:
> 
> https://wiki.openstack.org/wiki/Nova/CoreTeam#Review_Prioritization
> https://wiki.openstack.org/wiki/ReviewWorkflowTips
> 
> 3) Something much more drastic
> 
> At some point we may grow to the point where we have to adopt a new
> model to scale further.  I don't think we're too far off.  However, I
> think we could stand to grow a dedicated nova-core a bit larger before
> we have to take that step.  We have 19 people, and not all of those are
> active right now (meaning the team will likely shrink a bit soon).

I tend to focus the bulk of my review activity on the libvirt driver,
since that's where most of my knowledge is. I've recently done some
reviews outside this area to help reduce our backlog, but I'm not
so comfortable approving stuff in many of the general infrastructure
shared areas since I've not done much work on those areas of code.

I think Nova is large enough that it (mostly) beyond the scope of any
one person to know all areas of Nova code well enough todo quality
reviews. IOW, as we grow the nova-core team further, it may be worth
adding more reviewers who have strong knowledge of specific areas &
can focus their review energy in those areas, even if their review
count will be low when put in the context of nova as a whole.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Frustrations with review wait times

[Daniel P. Berrange]

On Tue, Aug 27, 2013 at 10:55:03AM -0400, Russell Bryant wrote:
> On 08/27/2013 10:43 AM, Daniel P. Berrange wrote:
> > I tend to focus the bulk of my review activity on the libvirt driver,
> > since that's where most of my knowledge is. I've recently done some
> > reviews outside this area to help reduce our backlog, but I'm not
> > so comfortable approving stuff in many of the general infrastructure
> > shared areas since I've not done much work on those areas of code.
> > 
> > I think Nova is large enough that it (mostly) beyond the scope of any
> > one person to know all areas of Nova code well enough todo quality
> > reviews. IOW, as we grow the nova-core team further, it may be worth
> > adding more reviewers who have strong knowledge of specific areas &
> > can focus their review energy in those areas, even if their review
> > count will be low when put in the context of nova as a whole.
> 
> I'm certainly open to that.
> 
> Another way I try to do this unofficially is give certain +1s a whole
> lot of weight when I'm looking at a patch.  I do this regularly when
> looking over patches to hypervisor drivers I'm not very familiar with.
> 
> Another thing we could consider is take this approach more officially.
> Oslo has started doing this for its incubator.  A maintainer of a part
> of the code not on oslo-core has their +1 treated as a +2 on that code.
> 
> http://git.openstack.org/cgit/openstack/oslo-incubator/tree/MAINTAINERS

Yes, just having a list of expert maintainers for each area of Nova
would certainly be helpful in identifying whose comments to place
more weight by, regardless of anything else we might do.

Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] live-snapshot/cloning of virtual machines

[Daniel P. Berrange]

On Tue, Aug 27, 2013 at 12:13:49PM -0400, Russell Bryant wrote:
> On 08/27/2013 12:04 PM, Tim Smith wrote:
> > 
> > On Tue, Aug 27, 2013 at 8:52 AM, Russell Bryant  > > wrote:
> > 
> > > What about publishing the API as blacklisted by default? This way it
> > > would be available only to users that enable it explicitly, while
> > still
> > > supporting the scenario described above.
> > 
> > It still makes no sense to me to merge an API for a feature that can't
> > be used.
> > 
> > 
> > While it's true that there won't be an in-tree driver that supports the
> > API for this release cycle, we have a commercial driver that supports it
> > (https://github.com/gridcentric/cobalt).
> > 
> > Having the API standardized in Havana would ensure that client support
> > is immediately available for our users as well as for the other
> > hypervisor vendors should they release a supporting driver in the next 9
> > months. I believe there is precedent for publishing a nova API for those
> > purposes.
> 
> IMO, to be the healthiest project we can be, we must focus on what code
> is actually a part of Nova.  If you'd like to submit your changes for
> inclusion into Nova, then we can talk.
> 
> What you are seeing here is a part of the pain of maintaining a fork.  I
> am not OK with shifting part of that burden on to the upstream project
> when it doesn't help the upstream project *at all*.
> 
> When we have supporting code to make the feature usable, then the API
> can go in.

Totally agreed with this. Supporting APIs in Nova with no in-tree users,
to satisfy the requirements of out of tree drivers should be an explicit
non-goal of the community IMHO. If a 3rd party does not wish to contribute
their code to Nova codebase, then it is expected that they take on all the
burden of doing the extra integration work their fork/branch implies.

For a code review POV, I would also not be satisfied doing review of APIs
without illustration of an in-tree driver wired up to the wire. Doing API
design is hard work, and I've been burnt too many times on other projects
adding APIs without in tree users, which then had to be thrown away or
replaced when the in-tree user finally arrived. So I would be very much
against adding any APIs without in-tree users, even ignoring the fact
that I think the "live VM cloning" concept as a whole is flawed.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Frustrations with review wait times

[Daniel P. Berrange]

On Wed, Aug 28, 2013 at 12:45:26PM +1000, Michael Still wrote:
> [Concerns over review wait times in the nova project]
> 
> I think that we're also seeing the fact that nova-core's are also
> developers. nova-core members have the same feature freeze deadline,
> and that means that to a certain extent we need to stop reviewing in
> order to get our own code ready by the deadline.
> 
> The strength of nova-core is that its members are active developers,
> so I think a "reviewer caste" would be a mistake. I am also not saying
> that nova-core should get different deadlines (although more leniency
> with exceptions would be nice).

Agreed, I think it is very important for the core reviewers to also be
active developers, since working on the code is how you gain the knowledge
required to do high quality reviews.

> So, I think lower review rates around deadlines are just a fact of life.

This is a fairly common problem across all open source projects really.
People consistently wait until just before review deadlines to submit
their code. You have to actively encourage people to submit their code
well before deadlines / discourage them from waiting till the last
minute. Sometimes the best way to get people to learn this is the hard
way, by postponing their feature if submitted too close to the dealine
and too much other stuff is ahead of it in the queue. IOW we should
prioritize review of work whose authors submitted earlier to encourage
good practice with early submission.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Frustrations with review wait times

[Daniel P. Berrange]

On Wed, Aug 28, 2013 at 03:43:21AM +, Joshua Harlow wrote:
> Why not a rotation though, I could see it beneficial to say have a
> group of active developers code for say a release then those
> developers rotate to a reviewer position only (and rotate again for
> every release). This allows for a flow of knowledge between reviewers
> and a different set of coders (instead of a looping flow since
> reviewers are also coders).
> 
> For a big project like nova the workload could be spread out more
> like that.

I don't think any kind of rotation system like that is really
practical. Core team members need to have the flexibility to balance
their various conflicting workloads in a way that maximises their
own productivity.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] New Nova hypervisor: Docker

[Daniel P. Berrange]

On Wed, Aug 28, 2013 at 06:00:50PM +1000, Michael Still wrote:
> On Wed, Aug 28, 2013 at 4:18 AM, Sam Alba  wrote:
> > Hi all,
> >
> > We've been working hard during the last couple of weeks with some
> > people. Brian Waldon helped a lot designing the Glance integration and
> > driver testing. Dean Troyer helped a lot on bringing Docker support in
> > Devstack[1]. On top of that, we got several feedback on the Nova code
> > review which definitely helped to improve the code.
> >
> > The blueprint[2] explains what Docker brings to Nova and how to use it.
> 
> I have to say that this blueprint is a fantastic example of how we
> should be writing design documents. It addressed almost all of my
> questions about the integration.

Yes, Sam (& any of the other Docker guys involved) have been great at
responding to reviewers' requests to expand their design document. The
latest update has really helped in understanding how this driver works
in the context of openstack from an architectural and functional POV.

> However, it would be nice for this to be on Launchpad, that being
> where we track blueprints. Would it be possible for you to move it
> over there? Or just link to the design doc from there?

Their blueprint on launchpad already links to the doc in fact, so no
change is needed

  https://blueprints.launchpad.net/nova/+spec/new-hypervisor-docker

links to

  
https://github.com/dotcloud/openstack-docker/blob/master/docs/nova_blueprint.md

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Frustrations with review wait times

[Daniel P. Berrange]

On Wed, Aug 28, 2013 at 10:29:23PM +1200, Robert Collins wrote:
> On 28 August 2013 21:13, Daniel P. Berrange  wrote:
> > On Wed, Aug 28, 2013 at 03:43:21AM +, Joshua Harlow wrote:
> 
> >> For a big project like nova the workload could be spread out more
> >> like that.
> >
> > I don't think any kind of rotation system like that is really
> > practical. Core team members need to have the flexibility to balance
> > their various conflicting workloads in a way that maximises their
> > own productivity.
> 
> So does everyone else, surely? Are you saying 'I don't think I can
> commit to regular reviewing', or are you saying 'all reviewers will be
> unable to commit to regular reviewing'? Or something else?

No, IIUC, Joshua was suggesting that core team members spend one cycle
doing reviews only, with no coding, and then reverse for the next cycle. 
That is just far too coarse/crude. Core team members need to be free to
balance their time between reviews and coding work on an ongoing basis,
just as any other member of the community can.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages

[Daniel P. Berrange]

On Tue, Aug 06, 2013 at 11:36:44PM -0300, Monty Taylor wrote:
> 
> 
> On 08/06/2013 11:14 PM, Robert Collins wrote:
> > On 7 August 2013 11:22, Jay Buffington  wrote:
> > 
> >> ln -s /usr/lib64/python2.6/site-packages/libvirtmod_qemu.so
> >> $(VENV)/lib/python2.6/site-packages/
> >>
> >> Why isn't libvirt-python on pypi?  AFAICT, nothing is stopping us from
> >> uploading it.  Maybe we should just stick it on there and this issue
> >> will be resolved once and for all.
> > 
> > Please please oh yes please :).
> 
> It doesn't build from a setup.py, so there is nothing to upload. It's
> built as part of the libvirt C library, and its build depends on scripts
> that autogenerate source code from the C library headers (think swig,
> except homegrown)

FYI, I have raised the issue of separating libvirt python into a
separate module for PyPI on the upstream libvirt mailing list. OpenStack
are not the only people asking for this, so I think it is inevitable
that libvirt upstream will start to offer the python binding on PyPI
in the future. eg it is a question of when, not if.

  https://www.redhat.com/archives/libvir-list/2013-August/msg01525.html


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] FFE Request: hyper-v-remotefx

[Daniel P. Berrange]

On Fri, Sep 06, 2013 at 10:56:10AM +, Alessandro Pilotti wrote:
> The RemoteFX feature allows Hyper-V compute nodes to provide GPU acceleration 
> to instances by sharing the host's GPU resources.
> 
> Blueprint: https://blueprints.launchpad.net/nova/+spec/hyper-v-remotefx
> 
> This feature provides big improvements for VDI related scenarios based on 
> OpenStack and Hyper-V compute nodes.
> It basically impacts on the driver code only plus an additional optional 
> scheduler filter.
> 
> A full architectural description has been added in the blueprint.
> 
> The patch has been published during H3 on Aug 18th and initially reviewed
> on Sept 4th with some very good ideas for improvements at a larger scale,
> subsequently implemented on the same day, unfortunately too late in the cycle.

Simply adding the blueprint description is not sufficient to remove
my objections. The patch as proposed is too incomplete to be merged IMHO.
I pointed out a number of design flaws in the review. The updated info in
the blueprint just re-inforces my review points, to further demonstrate
why this patch should not be merged as it is coded today. It will require
non-negliglable additional dev work to address this properly I believe,
so I'm afraid that I think this is not suitable material for Havana at
this point in time.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] FFE Request: hyper-v-remotefx

[Daniel P. Berrange]

On Fri, Sep 06, 2013 at 12:22:27PM +, Alessandro Pilotti wrote:
> 
> On Sep 6, 2013, at 14:26 , "Daniel P. Berrange" 
> mailto:berra...@redhat.com>>
>  wrote:
> 
>>  On Fri, Sep 06, 2013 at 10:56:10AM +, Alessandro Pilotti wrote:
>> The RemoteFX feature allows Hyper-V compute nodes to provide GPU 
>> acceleration to instances by sharing the host's GPU resources.
>>
>> Blueprint: https://blueprints.launchpad.net/nova/+spec/hyper-v-remotefx
>>
>> This feature provides big improvements for VDI related scenarios based on 
>> OpenStack and Hyper-V compute nodes.
>> It basically impacts on the driver code only plus an additional optional 
>> scheduler filter.
>> 
>> A full architectural description has been added in the blueprint.
>> 
>> The patch has been published during H3 on Aug 18th and initially reviewed
>> on Sept 4th with some very good ideas for improvements at a larger scale,
>> subsequently implemented on the same day, unfortunately too late in the 
>> cycle.
> 
> Simply adding the blueprint description is not sufficient to remove
> my objections. The patch as proposed is too incomplete to be merged IMHO.
> I pointed out a number of design flaws in the review. The updated info in
> the blueprint just re-inforces my review points, to further demonstrate
> why this patch should not be merged as it is coded today. It will require
> non-negliglable additional dev work to address this properly I believe,
> so I'm afraid that I think this is not suitable material for Havana at
> this point in time.
> 
> I already committed an updated patch after your review on the 4th addressing
> your observations, agreeing that the initial implementation was missing
> flexibility (I only didn't commit the scheduler filter yet, as IMO it
> requires a separate dependent patch, but this can be done anytime).

IMHO the lack of schedular support is a blocker item. Running a VM with
this feature requires that the scheduler be able to place the VM on a
host which supports the feature. Without this, users are just relying on
lucky placement when trying to boot a VM to use this feature.

> To allow others to add their opinion easily, I'm following up here to 
> your objections in the blueprint which basically can be reduced (correct
> me if I'm wrong) to the following positions, beside the areas on which
> we already agreed regarding scheduling and which have already been
> implemented:
> 
> 1) You suggest to define the amount of RemoteFX GPU resources required in the 
> flavour
> 2) I suggest to provide those requirements in the image custom properties 
> (which is how it is currently implemented, see blueprint description as well).
> 
> Beside the fact that the two solutions are IMO not mutually exclusive
> as scheduling filters could simply apply both, the reason why I don't
> see how flavours should be considered for this patch is simple:

IIUC, this feature consumes finite host & network resources. Allowing
users to request it via the image properties is fine, but on its own
I consider that insecure. The administrator needs to be able to control
how can access these finite resources, in the same way that you don't
allow users to specify arbitrary amounts of memory, or as many vCPUS
as they want. It seems that flavours are the only place to do this.
Again, I consider this a blocker, not something to be tacked on later.

> AFAIK Nova flavors at the moment don't support custom properties
> (please correct me if I'm wrong here, I looked at the flavors APIs
> and implementation, but I admit my ignorance in the flavor internals), so

I'm honestly not sure, since I don't know enough about the flavours
APIs.

> 1) Adding the remotefx requisites at the system metadata level 
>https://github.com/openstack/nova/blob/master/nova/compute/flavors.py#L60
>would be IMO wrong, as it would tightly couple a hypervisor specific limit
>with a generic compute API.
> 2) Adding custom properties to flavors goes way beyond the scope of this 
> blueprint.
> 
> From an administrative and ACL perspective, administrators can selectively 
> provide access to users to a given image, thus limiting the access to 
> RemoteFX GPU resources (video memory in primis).
> 
> Being able to do it ALSO at the flavour level would add additional 
> granularity,
> e.g. assigning different screen resolutions, using a single image. I 
> personally
> see this as a separate blueprint as it impacts on the design of a fundamental
> Nova feature that will need quite some discussion.

That is quite likely/possibly correct.

That we're having this level of design debate, is exactly why I think this
is not suitable

Re: [openstack-dev] [Nova] FFE Request: oslo-messaging

[Daniel P. Berrange]

On Fri, Sep 06, 2013 at 09:49:18AM -0400, Russell Bryant wrote:
> On 09/06/2013 08:30 AM, Mark McLoughlin wrote:
> > On Fri, 2013-09-06 at 10:59 +0200, Thierry Carrez wrote:
> >> Mark McLoughlin wrote:
> >>> I'd like to request a feature freeze exception for the final (and
> >>> admittedly the largest) patch in the series of 40 patches to port Nova
> >>> to oslo.messaging:
> >>>
> >>>   https://review.openstack.org/39929
> >>
> >> I'm generally adverse to granting feature freeze exceptions to code
> >> refactoring: the user benefit of having them in the release is
> >> inexistent, while they introduce some risk by changing deep code
> >> relatively late in the cycle. That's why I prefer those to be targeted
> >> to earlier development milestones, this avoids having to make hard calls
> >> once all the work is done and almost completed.
> > 
> > Yes, absolutely understood.
> > 
> > To be clear - while I think there's a strong case for an exception here,
> > I am very close to this, so I would be cool with a denial of this
> > request.
> > 
> >> That said, if the risk is under control and the patch is ready to merge,
> >> I'm fine with this as long as there is some other benefits in having it
> >> *in* the release rather than landed first thing in icehouse.
> >>
> >> Would having it *in* the release facilitate stable/havana branch
> >> maintenance, for example ?
> >>
> >>> While this change doesn't provide any immediate user-visible benefit, it
> >>> would be massively helpful in maintaining momentum behind the effort all
> >>> through the Havana cycle to move the RPC code from oslo-incubator into a
> >>> library.
> >>
> >> Could you expand on why this would be a lot more helpful to have it in
> >> the release rather than early in icehouse ?
> >>
> >> And to have all cards on the table, how much sense would the alternative
> >> make (i.e. not land this final patch while a lot of this feature code
> >> has already been merged) ?
> > 
> > If the patch was merged now, while it's not a user-visible feature
> > per-se, I think oslo.messaging is something we would celebrate in the
> > Havana release e.g.
> > 
> >   While OpenStack continues to add new projects and developers while, 
> >   in parallel, aggressively takes steps to enable the project to scale. 
> >   The oslo.messaging library added in the Havana release is an example 
> >   of where code which was previously copied and pasted between projects 
> >   and has now been re-factored out into a shared library with a clean 
> >   API. This library will provide a structured way for OpenStack 
> >   projects to collaborate on adopting new messaging patterns and 
> >   features without the disruption of incompatible API changes nor the 
> >   pain of keeping copied and pasted code in sync.
> > 
> > Obviously, as Oslo PTL, I think this is an important theme that we
> > should continue to emphasise and build momentum around. The messaging
> > library is by far the most significant example so far of how this
> > process of creating libraries can be effective. Nova using the library
> > in the Havana release would (IMHO) be the signal that this process is
> > working and hopefully inspire others to take a similar approach with
> > other chunks of common code.
> > 
> > Conversely, if we delayed merging this code until Icehouse, I think it
> > would leave somewhat of a question mark hanging over oslo.messaging and
> > Oslo going into the summit:
> > 
> >   Is this oslo.messaging thing for real? Or will it be abandoned and we
> >   need to continue with the oslo-incubator RPC code? Why is it taking so
> >   long to create these libraries? This sucks!

I think you're under-selling yourself here. As an interested 3rd party to
oslo development, that certainly isn't my impression of what has happened
with oslo.messaging development until this point.

I think you have a pretty credible story to tell about the work done to
get oslo.messaging to where it is now for Havana, even without it being
used by Nova & don't think anyone could credibly claim it is dead or
moving too slowly. Reusable library design is hard to get right & takes
time if you want to support a stable API long term. 

I don't know about your non-Nova plans, but doing the final conversion in
Icehouse timeframe may give you time in which to convert other openstack
projects to oslo.messaging at the same time, so we would have everything
aligned at once.

> > That's all very meta, I know. But I do really believe making progress
> > with Oslo libraries is important for OpenStack long-term. While it's not
> > a user-visible benefit per-se, I do think this work benefits the project
> > broadly and is also something worth marketing.
> > 
> > We measure our progress in terms of what we achieved in each release
> > cycle. I think we've made great progress on oslo.messaging in Havana,
> > but unless a project uses it in the release it won't be something we
> > really celebrate until Icehouse.

I can understand where

Re: [openstack-dev] [Nova] FFE Request: utilization aware scheduling

[Daniel P. Berrange]

On Fri, Sep 06, 2013 at 01:07:49PM +0200, Nikola Đipanov wrote:
> On 06/09/13 11:28, Thierry Carrez wrote:
> > Wang, Shane wrote:
> >> Hi core developers and everyone,
> >>
> >> Please allow me to make an FFE request for adding utilization aware 
> >> scheduling support in Havana.
> >>
> >> The blueprint: 
> >> https://blueprints.launchpad.net/nova/+spec/utilization-aware-scheduling.
> >> [...]
> > 
> > This is a bit in the same bucket as the hyper-v-rdp-console above...
> > It's a significant feature but it's not critical to the success of the
> > Havana integrated release (doesn't affect other projects), and it looks
> > like it still needs a significant review effort before it can be merged.
> > 
> 
> There seems to be a pretty fundamental disagreement on the approach of
> the first patch in the series, and it was stated several times[1] that
> the issue needs consensus that we were hoping to reach on the summit or
> on the ML and that it was too late in the H cycle to be proposing such
> changes.
> 
> With this in mind (unless I am missing something, of course) I don't see
> how we could suddenly decide to merge this. I don't think this is
> related to code quality in any way, as Shane seems to imply in the first
> message, but the lack of agreement on the approach.

Agreed, if there are unresolved design questions, this is a strong
sign that it should wait for the next cycle.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra][all] Reviews with a prio label?

[Daniel P. Berrange]

On Tue, Oct 20, 2015 at 06:09:51PM +0200, Markus Zoeller wrote:
> In ML post [1] I wondered if it would be possible to introduce a new
> "prio" label in Gerrit which could help in focusing review efforts to
> increase the throughput. With this new post I'd like to discuss if we
> think this could be useful. For example, this would allow to create this
> query in Gerrit:
> 
> "status:open label:Prio=3" 
> 
> I was curious how this could look like in Gerrit, which resulted in the
> screenshots available at [2]. This would minimize the gap between the 
> prio of the blueprints/bugs and their commit reviews.
> 
> I'm mostly active in Nova, so here a short example of how we currently
> try to speed up the merges of trivial fixes:
> 
> * contributor "A" spots a review which looks trivial
> * contributor "A" copies the review ID into an etherpad
> * core reviewer "B" reads the etherpad when possible
> * core reviewer "B" does a review too and eventually gives a +W
> * core reviewer "B" removes that review from the Etherpad when it merges
> 
> This workflow is only necessary because Gerrit does not allow to 
> categorize reviews, e.g. into a group of "trivial fixes". 
> 
> I noticed in my "mini poc" that it would be possible to set permissions
> to specific label values. Which allows us to have a "trivialfix" prio 
> which can be set by everyone, but also a "high" prio which can be set 
> by an automated entity which reuses the priority of the blueprint or 
> bug report.
> 
> Do you think this would speed things up? Or is this just nitpicking on
> an already good enough workflow?

What you're describing is really just a special-case of allowing
arbitrary user tagging of changes. If gerrit had a free-format
keyword tag facility that users could use & query, it'd open up many
possible options for improving our general workflow, and letting
users customize their own workflow.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra] Upgrade to Gerrit 2.11

[Daniel P. Berrange]

On Tue, Oct 13, 2015 at 05:08:29PM -0700, Zaro wrote:
> Hello All,
> 
> The openstack-infra team would like to upgrade from our Gerrit 2.8 to
> Gerrit 2.11.  We are proposing to do the upgrade shortly after the
> Mitaka summit.  The main motivation behind the upgrade is to allow us
> to take advantage of some of the new REST api, ssh commands, and
> stream events features.  Also we wanted to stay closer to upstream so
> it will be easier to pick up more recent features and fixes.

Looking at the release notes I see my most wanted new feature, keyword
tagging of changes, is available

[quote]
Hashtags.

Hashtags can be added to changes. The hashtags are stored in git notes and
are indexed in the secondary index.

This feature requires the notedb to be enabled.
[/quote]

It is listed as an experimental feature, but I'd really love to see this
enabled if at all practical.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][policy] Exposing hypervisor details to users

[Daniel P. Berrange]

On Fri, Nov 06, 2015 at 05:08:59PM +1100, Tony Breeds wrote:
> Hello all,
> I came across [1] which is notionally an ironic bug in that horizon 
> presents
> VM operations (like suspend) to users.  Clearly these options don't make sense
> to ironic which can be confusing.
> 
> There is a horizon fix that just disables migrate/suspened and other 
> functaions
> if the operator sets a flag say ironic is present.  Clealy this is sub optimal
> for a mixed hv environment.
> 
> The data needed (hpervisor type) is currently avilable only to admins, a quick
> hack to remove this policy restriction is functional.
> 
> There are a few ways to solve this.
> 
>  1. Change the default from "rule:admin_api" to "" (for 
> os_compute_api:os-extended-server-attributes and
> os_compute_api:os-hypervisors), and set a list of values we're
> comfortbale exposing the user (hypervisor_type and
> hypervisor_hostname).  So a user can get the hypervisor_name as part of
> the instance deatils and get the hypervisor_type from the
> os-hypervisors.  This would work for horizon but increases the API load
> on nova and kinda implies that horizon would have to cache the data and
> open-code assumptions that hypervisor_type can/can't do action $x
> 
>  2. Include the hypervisor_type with the instance data.  This would place the 
> burdon on nova.  It makes the looking up instance details slightly more
> complex but doesn't result in additional API queries, nor caching
> overhead in horizon.  This has the same opencoding issues as Option 1.
> 
>  3. Define a service user and have horizon look up the hypervisors details 
> via 
> that role.  Has all the drawbacks as option 1 and I'm struggling to
> think of many benefits.
> 
>  4. Create a capabilitioes API of some description, that can be queried so 
> that
> consumers (horizon) can known
> 
>  5. Some other way for users to know what kind of hypervisor they're on, 
> Perhaps
> there is an established image property that would work here?
> 
> If we're okay with exposing the hypervisor_type to users, then #2 is pretty
> quick and easy, and could be done in Mitaka.  Option 4 is probably the best
> long term solution but I think is best done in 'N' as it needs lots of
> discussion.

I think that exposing hypervisor_type is very much the *wrong* approach
to this problem. The set of allowed actions varies based on much more than
just the hypervisor_type. The hypervisor version may affect it, as may
the hypervisor architecture, and even the version of Nova. If horizon
restricted its actions based on hypevisor_type alone, then it is going
to inevitably prevent the user from performing otherwise valid actions
in a number of scenarios.

IMHO, a capabilities based approach is the only viable solution to
this kind of problem.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][policy] Exposing hypervisor details to users

[Daniel P. Berrange]

On Fri, Nov 06, 2015 at 07:09:59AM -0500, Sean Dague wrote:
> On 11/06/2015 04:49 AM, Daniel P. Berrange wrote:
> > 
> > I think that exposing hypervisor_type is very much the *wrong* approach
> > to this problem. The set of allowed actions varies based on much more than
> > just the hypervisor_type. The hypervisor version may affect it, as may
> > the hypervisor architecture, and even the version of Nova. If horizon
> > restricted its actions based on hypevisor_type alone, then it is going
> > to inevitably prevent the user from performing otherwise valid actions
> > in a number of scenarios.
> > 
> > IMHO, a capabilities based approach is the only viable solution to
> > this kind of problem.
> 
> Right, we just had a super long conversation about this in #openstack-qa
> yesterday with mordred, jroll, and deva around what it's going to take
> to get upgrade tests passing with ironic.
> 
> Capabilities is the right approach, because it means we're future
> proofing our interface by telling users what they can do, not some
> arbitrary string that they need to cary around a separate library to
> figure those things out.
> 
> It seems like capabilities need to exist on flavor, and by proxy instance.
> 
> GET /flavors/bm.large/capabilities
> 
> {
>  "actions": {
>  'pause': False,
>  'unpause': False,
>  'rebuild': True
>  ..
>   }
> 
> A starting point would definitely be the set of actions that you can
> send to the flavor/instance. There may be features beyond that we'd like
> to classify as capabilities, but actions would be a very concrete and
> attainable starting point. With microversions we don't have to solve
> this all at once, start with a concrete thing and move forward.

I think there are two distinct use cases for capabilities we need to
consider.

 1. Before I launch an instance, does the cloud provide features XYZ

 2. Given this running instance, am I able to perform operation XYZ

Having capabilities against the flavour /might/ be sufficient for
#1, but it isn't sufficient for #2.

For example, the ability to hotplug disks to a running instance will
depend on what disk controller the instance is using. The choice of
disk controller used will vary based on image metadata properties,
eg ide vs scsi vs virtio-blk. IDE does not support hotplug, but
scsi & virtio-blk do. So we can't answer the question "does hotplug
disk work for this instance" simply based on the flavour - we need
to ask it against the instance.

What we can answer against the flavour is whether the hypervisor
driver is able to support hotplug in principle, given a suitably
configured instance. That said, even that is not an exact science
if you take into account fact that the cloud could be running
compute nodes with different versions, and the flavour does not
directly control which version of a compute node we'll run against.

Having capabilities against the flavour would certainly allow for
an improvement in Horizon UI vs its current state, but to be able
to perfectly represent what is possible for an instance, Horizon
would ultimately require capabilities against the isntance,

So I think we'll likely end up having to implement both capabilities
against a flavour and against an instance. So you'd end up with a
flow something like

 - Check to see if cloud provider supports hotplug by checking
   flavour capability==disk-hotplug

 - When booting an instance mark it as requiring capability==disk-hotplug
   to ensure its scheduled to a node which supports that capability

 - When presenting UI for operations against an instance, check
   that the running instance supports capability==disk-hotplug


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Proposal to add Sylvain Bauza to nova-core

[Daniel P. Berrange]

On Fri, Nov 06, 2015 at 03:32:00PM +, John Garbutt wrote:
> Hi,
> 
> I propose we add Sylvain Bauza[1] to nova-core.
> 
> Over the last few cycles he has consistently been doing great work,
> including some quality reviews, particularly around the Scheduler.
> 
> Please respond with comments, +1s, or objections within one week.

+1 from me, I think Sylvain will be a valuable addition to the team
for his scheduler expertize.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Proposal to add Alex Xu to nova-core

[Daniel P. Berrange]

On Fri, Nov 06, 2015 at 03:32:04PM +, John Garbutt wrote:
> Hi,
> 
> I propose we add Alex Xu[1] to nova-core.
> 
> Over the last few cycles he has consistently been doing great work,
> including some quality reviews, particularly around the API.
> 
> Please respond with comments, +1s, or objections within one week.

+1 from me, the tireless API patch & review work has been very helpful
to our efforts in this area.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [tc][all][osprofiler] OSprofiler is dead, long live OSprofiler

[Daniel P. Berrange]

On Mon, Nov 09, 2015 at 02:57:37AM -0800, Boris Pavlovic wrote:
> Hi stackers,
> 
> Intro
> ---
> 
> It's not a big secret that OpenStack is huge and complicated ecosystem of
> different
> services that are working together to implement OpenStack API.
> 
> For example booting VM is going through many projects and services:
> nova-api, nova-scheduler, nova-compute, glance-api, glance-registry,
> keystone, cinder-api, neutron-api... and many others.
> 
> The question is how to understand what part of the request takes the most
> of the time and should be improved. It's especially interested to get such
> data under the load.
> 
> To make it simple, I wrote OSProfiler which is tiny library that should be
> added to all OpenStack
> projects to create cross project/service tracer/profiler.
> 
> Demo (trace of CLI command: nova boot) can be found here:
> http://boris-42.github.io/ngk.html
> 
> This library is very simple. For those who wants to know how it works and
> how it's integrated with OpenStack take a look here:
> https://github.com/openstack/osprofiler/blob/master/README.rst
> 
> What is the current status?
> ---
> 
> Good news:
> - OSprofiler is mostly done
> - OSprofiler is integrated with Cinder, Glance, Trove & Ceilometer
> 
> Bad news:
> - OSprofiler is not integrated in a lot of important projects: Keystone,
> Nova, Neutron
> - OSprofiler can use only Ceilometer + oslo.messaging as a backend
> - OSprofiler stores part of arguments in api-paste.ini part in project.conf
> which is terrible thing
> - There is no DSVM job that check that changes in OSprofiler don't break
> the projects that are using it
> - It's hard to enable OSprofiler in DevStack
> 
> Good news:
> I spend some time and made 4 specs that should address most of issues:
> https://github.com/openstack/osprofiler/tree/master/doc/specs
> 
> Let's make it happen in Mitaka!
> 
> Thoughts?
> By the way somebody would like to join this effort?)

I'm very interested in seeing this kind of capability integrated across
openstack. I've really needed it in working in Nova for many times.
6 months back or so (when I didn't know osprofiler existed), I hacked up
a roughly equivalent library for openstack:

   https://github.com/berrange/oslo.devsupport

I never had time to persue this further and then found out about osprofiler
so it dropped in priority for me.

Some notable things I think I did differently

 - Used oslo.versionedobjects for recording the data to provide a
   structured data model with easy support for future extension,
   and well defined serialization format

 - Structured data types for all the various common types of operation
   to instrument (database request, RPC call, RPC dispatch, REST call
   REST dispatch, native library call, thread spawn, thread main,
   external command spawn). This is to ensure all apps using the library
   provide the same set of data for each type of operation.

 - Ability to capture stack traces against each profile point to
   allow creation of control flow graphs showing which code paths
   consumed significant time.

 - Abstract "consumer" API for different types of persistence backend.
   Rather than ceilometer, my initial consumer just serialized to
   plain files in well known directories, using oslo.versionedobjects
   serialization format. I can see ceilometer might be nice for
   production deployments, but plain files was simpler for developer
   environments which might not even be running ceilometer

 - Explicit tracking of nesting of instrunmented operation had a parent
   operation. At the top level was things like thread main, RPC dispatch
   and REST dispatch. IIUC, with osprofiler you could potentially infer
   the nesting by sorting based on start/end timestamps, but I felt an
   explicit representation was nicer to work with from the object
   model POV.

My approach would require oslo.devsupport to be integrated into the
oslo.concurrency, oslo.db, oslo.messaging components, as well as the
various apps. I did quick hacks to enable this for Nova & pieces it
uses you can see from these (non-upstreamed) commits:

  
https://github.com/berrange/oslo.concurrency/commit/2fbbc9cf4f23c5c2b30ff21e9e06235a79edbc20
  
https://github.com/berrange/oslo.messaging/commit/5a1fd87650e56a01ae9d8cc773e4d030d84cc6d8
  
https://github.com/berrange/nova/commit/3320b8957728a1acf786296eadf0bb40cb4df165

I see you already intend to make ceilometer optional and allow other
backends, so that's nice. I would love to see Osprofiler take on some
of the other ideas I had in my alternative, particularly the data model
based around oslo.versionedobjects to provide standard format for various
core types of operation, and the ability to record stack traces.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http:/

Re: [openstack-dev] [nova] hackathon day

[Daniel P. Berrange]

On Thu, Nov 12, 2015 at 11:15:09AM +, Rosa, Andrea (HP Cloud Services) 
wrote:
> Hi
> 
> I knew that people in China had a 3 days hackathon  few months ago I was 
> thinking to have a similar thing in Europe.
> My original idea was to propose to add an extra day after the mid-cycle but I 
> am not sure if that is a good idea anymore:

The day after the mid-cycle is the main day for travelling to FOSDEM, so
a bad choice for people who want to attend FOSDEM too.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] [nova] Image Signature Verification

[Daniel P. Berrange]

On Thu, Nov 12, 2015 at 08:30:53PM +, Poulos, Brianna L. wrote:
> Hello,
> 
> There has recently been additional discussion about the best way to handle
> image signature verification in glance and nova [1].  There are two
> options being discussed for the signature (the examples below using
> 'RSA-PSS' as the type, and SHA-256 as the hash method):
> 
> 1. The signature is of the glance checksum of the image data (currently a
> hash which is hardcoded to be MD5)
> signature = RSA-PSS(SHA-256(MD5(IMAGE-CONTENT)))
> 
> 2. The signature of the image data directly
> signature = RSA-PSS(SHA-256(IMAGE-CONTENT))
> 
> The 1st option is what is currently in glance's liberty release [2].  This
> approach was chosen with the understanding that the glance checksum would
> be updated to be configurable [3].  Although the 2nd option was initially
> proposed, the glance community opposed it during the pre-Liberty virtual
> mini-summit in May 2015 (due to the performance impact of doing two hashes
> of the image data--one for the 'checksum' and the other for the
> signature), and it was decided to proceed with the 1st option during the
> Liberty summit [4].
> 
> During the Mitaka Summit, making the glance checksum configurable was
> discussed during a design session [5].  It was decided that instead of
> making the 'checksum' image field configurable, it would be preferable to
> compute a separate, configurable (on a per-image basis, with a site-wide
> default) hash, and then use that hash when MD5 wasn't sufficient (such as
> in the case of signature verification). This second hash would be computed
> at the same time the MD5 'checksum' was computed.
> 
> Which brings us to the nova spec which is under discussion [1], which is
> to add the ability to verify signatures in nova.  The nova community has
> made it clear that the promise of providing a configurable hash in glance
> is not good enough--they never want to support any signatures that use MD5
> in any way, shape, or form; nor do they want to rely on asking glance for
> what hash option was used.  To that end, the push is to use the 2nd option
> to verify signatures in nova from the start.

As well as not wanting MD5, I believe that computing signatures based
on a configurable checksum in glance provides a bad user experiance.
The user generating the signature of their image, now has to have a
way to query glance to find out what checksum it used, in order to
generate their signature. Further if the glance admin ever wants to
change their checksum algorithm, they'd break all existing signatures
by doing so. These are just as important reasons why I want Nova
to use the 2nd option and compute signatures directly on the image
content.

> Since the glance community no longer seems opposed to the idea of
> computing two hashes (the second hash being optional, of course), the 2nd
> option has now become valid from the glance perspective.  This would
> require modifying the existing implementation in glance to verify a
> signature of the image data, rather than verifying a checksum of the image
> data, but would have no additional performance hit beyond the cost to
> compute the second hash.  Note that the image data would still only be
> read once -- the checksum update (for the MD5 hash) and the signature
> verification update (for the signature hash) would occur in the same loop.
> Although this would mean that signatures generated using option 1 would no
> longer verify, since signatures generated using option 1 are based on an
> MD5 hash (and were waiting for the checksum configurability before
> becoming a viable cryptographic option anyway), this does not pose a
> significant issue.

A second point about the current proposal from Nova's POV is that
we do not like the image property names currently used. In Liberty
Nova standardized on the property naming scheme it uses to have 3
naming prefixes

  https://github.com/openstack/nova/blob/master/nova/objects/image_meta.py#L166

 - 'hw_' - properties that affect virtual hardware configuration
 - 'os_' - properties that affect guest OS setup / configuration
 - 'img_' - properties that affect handling of images by the host

The signature properties are obviously all related to the handling
of images by the host, so from Nova's POV we should have an 'img_'
prefix on all their names.

We probably should have alerted glance devs to this naming convention
before now to avoid this problem, but I guess we forgot. It would be
great if glance devs could bear this preferred naming convention in
mind if there are any future cases where there is a property that
needs to be used by both Nova & Glance code.

Anyway since the change in the way we calculate signatures on images
is a non-backwards compatible change for users of the current glance
impl, changing these property names at this point is reasonable todo.

Glance could use the property name to determine whether it is
getting an old or new style signature. ie if the image

Re: [openstack-dev] [glance] [nova] Image Signature Verification

[Daniel P. Berrange]

On Tue, Nov 17, 2015 at 02:09:42PM -0300, Flavio Percoco wrote:
> On 13/11/15 09:35 +0000, Daniel P. Berrange wrote:
> >On Thu, Nov 12, 2015 at 08:30:53PM +, Poulos, Brianna L. wrote:
> >>Hello,
> >>
> >>There has recently been additional discussion about the best way to handle
> >>image signature verification in glance and nova [1].  There are two
> >>options being discussed for the signature (the examples below using
> >>'RSA-PSS' as the type, and SHA-256 as the hash method):
> >>
> >>1. The signature is of the glance checksum of the image data (currently a
> >>hash which is hardcoded to be MD5)
> >>signature = RSA-PSS(SHA-256(MD5(IMAGE-CONTENT)))
> >>
> >>2. The signature of the image data directly
> >>signature = RSA-PSS(SHA-256(IMAGE-CONTENT))
> >>
> >>The 1st option is what is currently in glance's liberty release [2].  This
> >>approach was chosen with the understanding that the glance checksum would
> >>be updated to be configurable [3].  Although the 2nd option was initially
> >>proposed, the glance community opposed it during the pre-Liberty virtual
> >>mini-summit in May 2015 (due to the performance impact of doing two hashes
> >>of the image data--one for the 'checksum' and the other for the
> >>signature), and it was decided to proceed with the 1st option during the
> >>Liberty summit [4].
> >>
> >>During the Mitaka Summit, making the glance checksum configurable was
> >>discussed during a design session [5].  It was decided that instead of
> >>making the 'checksum' image field configurable, it would be preferable to
> >>compute a separate, configurable (on a per-image basis, with a site-wide
> >>default) hash, and then use that hash when MD5 wasn't sufficient (such as
> >>in the case of signature verification). This second hash would be computed
> >>at the same time the MD5 'checksum' was computed.
> >>
> >>Which brings us to the nova spec which is under discussion [1], which is
> >>to add the ability to verify signatures in nova.  The nova community has
> >>made it clear that the promise of providing a configurable hash in glance
> >>is not good enough--they never want to support any signatures that use MD5
> >>in any way, shape, or form; nor do they want to rely on asking glance for
> >>what hash option was used.  To that end, the push is to use the 2nd option
> >>to verify signatures in nova from the start.
> >
> >As well as not wanting MD5, I believe that computing signatures based
> >on a configurable checksum in glance provides a bad user experiance.
> >The user generating the signature of their image, now has to have a
> >way to query glance to find out what checksum it used, in order to
> >generate their signature. Further if the glance admin ever wants to
> >change their checksum algorithm, they'd break all existing signatures
> >by doing so. These are just as important reasons why I want Nova
> >to use the 2nd option and compute signatures directly on the image
> >content.
> 
> This is a very good point. Thanks for bringing it up, Dan.
> 
> >
> >>Since the glance community no longer seems opposed to the idea of
> >>computing two hashes (the second hash being optional, of course), the 2nd
> >>option has now become valid from the glance perspective.  This would
> >>require modifying the existing implementation in glance to verify a
> >>signature of the image data, rather than verifying a checksum of the image
> >>data, but would have no additional performance hit beyond the cost to
> >>compute the second hash.  Note that the image data would still only be
> >>read once -- the checksum update (for the MD5 hash) and the signature
> >>verification update (for the signature hash) would occur in the same loop.
> >>Although this would mean that signatures generated using option 1 would no
> >>longer verify, since signatures generated using option 1 are based on an
> >>MD5 hash (and were waiting for the checksum configurability before
> >>becoming a viable cryptographic option anyway), this does not pose a
> >>significant issue.
> >
> >A second point about the current proposal from Nova's POV is that
> >we do not like the image property names currently used. In Liberty
> >Nova standardized on the property naming scheme it uses to have 3
> >naming prefixes
> >
> > https://github.com/openstack/nova/blob/master/nova/objects/image_meta.py#L166

Re: [openstack-dev] [nova][infra] Getting a bleeding edge libvirt gate job running

[Daniel P. Berrange]

On Wed, Nov 18, 2015 at 05:18:28PM +1100, Tony Breeds wrote:
> On Tue, Nov 17, 2015 at 03:32:45PM -0800, Jay Pipes wrote:
> > On 11/17/2015 11:10 AM, Markus Zoeller wrote:
> > >Background
> > >==
> > >The blueprint [1] wants to utilize the *virtlogd* logging deamon from
> > >libvirt. Among others to solve bug [2], one of our oldest ones. The
> > >funny part is, that this libvirt feature is still in development. This
> > >was a trigger to see if we can create a gate job which utilizes the
> > >latest, bleeding edge, version of libvirt to test such features. We
> > >discussed it shortly in IRC [3] (tonyb, bauzas, markus_z) and wanted to
> > >get some feedback here. The summary of the idea is:
> > >* Create a custom repo which contains the latest libvirt version
> > >* Enhance Devstack so that it can point to a custom repo to install
> > >   the built libvirt packages
> > >* Have a nodepool image which is compatible with the libvirt packages
> > >* In case of [1]: check if tempest needs further/changed tests
> > >
> > >Open questions
> > >==
> > >* Is already someone working on something like that and I missed it?
> > 
> > Sean (cc'd) might have some information on what he's doing in the OVS w/
> > DPDK build environment, which AFAIK requires a later build of libvirt than
> > available in most distros.
> > 
> > >* If 'no', is there already a repo which contains the very latest
> > >   libvirt builds which we can utilize?
> > >
> > >I haven't done anything with the gates before, which means there is a
> > >very high chance I'm missing something or missunderstanding a concept.
> > >Please let me know what you think.
> > 
> > A generic "build libvirt or OVS from this source repo" dsvm job would be
> > great I think. That would allow overrides in ENV variables to point the job
> > to a URI for grabbing sources of OVS (DPDK OVS, mainline OVS) or libvirt
> > that would be built into the target nodepool images.
> 
> I was really hoping to decouple the build from the dsvm jobs.  My initial
> thoughts were a add a devstack plugin that add $repo and then upgrade
> $packages.  I wanted to decouple the build from install as I assumed that the
> delays in building libvirt (etc) would be problematic *and* provide another
> failure mode for devstack that we really don't want to deal with.
> 
> I was only thinking of having libvirt and qemu in there but if the plug-in was
> abstract enough it could easily provide packages for other help utils (like 
> OVS
> and DPDK).
> 
> When I started looking at this Ubuntu was the likely candidate as Fedora in 
> the gate
> wasn't really a stable thing.  I see a little more fedora in nodepool so 
> perhaps a
> really quick win would be to just use the lib-virt preview on F22.

Trying to build from bleeding edge is just a can of worms as you'll need to
have someone baby-sitting the job to fix it up on new releases when the
list of build deps changes or build options alter. As an example, next
QEMU release will require you to pull in 3rd party libcacard library
for SPICE build, since it was split out, so there's already a build
change pending that would cause a regression in the gate.

So, my recommendation would really be to just use Fedora with virt-preview
for the bleeding edge and avoid trying to compile stuff in the gate. The
virt-preview repository tracks upstream releases of QEMU+Libvirt+libguestfs
with minimal delay and is built with the same configuration as future Fedora
releases will use. So such testing is good evidence that Nova won't break on
the next Fedora release.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Infra] Remove .mailmap files from OpenStack repos

[Daniel P. Berrange]

On Thu, Nov 19, 2015 at 07:28:03PM +0300, Mikhail Fedosin wrote:
> Currently we have .mailmap files in the root of almost all OpenStack repos:
> https://github.com/openstack/glance/blob/master/.mailmap
> https://github.com/openstack/horizon/blob/master/.mailmap
> https://github.com/openstack/nova/blob/master/.mailmap
> https://github.com/openstack/cinder/blob/master/.mailmap
> etc.
> 
> But it seems that they're very outdated (last time many of them were
> updated about 2 years ago). So, do we still need these files or it's better
> to remove them?

I think that the mailmap files provide pretty useful feature when
looking at the git history to see who wrote what. They ensure you
are given the best current email address for the author, not whatever
(now bouncing) email address they had 4 years ago. So deleting them
is not a great idea IMHO.

There could be better ways to maintain them though. Given that it
is not unusual for people to work across multiple different openstack
projects, it seems silly to manually update .mailmap in each project
individually.

We could have a single globally maintained mailmap file which we
automatically sync out to each project's GIT repo. It probably
wouldn't be too hard to write a script that looked at the list
of GIT authors in history and identify a large portion of the
cases where someone has changed their email addr and so let us
semi-automatically update the central mailmap file.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][nova]Move encryptors to os-brick

[Daniel P. Berrange]

On Fri, Nov 20, 2015 at 03:22:04AM +, Li, Xiaoyan wrote:
> Hi all,
> 
> To fix bug [1][2] in Cinder, Cinder needs to use nova/volume/encryptors[3]
> to attach/detach encrypted volumes. 
> 
> To decrease the code duplication, I raised a BP[4] to move encryptors to
> os-brick[5].
> 
> Once it is done, Nova needs to update to use the common library. This
> is BP raised. [6]

You need to proposal a more detailed spec for this, not merely a blueprint
as there are going to be significant discussion points here.

In particular for the QEMU/KVM nova driver, this proposal is not really
moving in a direction that is aligned with our long term desire/plan for
volume encryption and/or storage management in Nova with KVM.  While we
currently use dm-crypt with volumes that are backed by block devices,
this is not something we wish to use long term. Increasingly the storage
used is network based, and while we use in-kernel network clients for
iSCSI/NFS, we use an in-QEMU client for RBD/Gluster storage. QEMU also
has support for in-QEMU clients for iSCSI/NFS and it is likely we'll use
them in Nova in future too.

Now encryption throws a (small) spanner in the works as the only way to
access encrypted data right now is via dm-crypt, which obviously doesn't
fly when there's no kernel block device to attach it to. Hence we are
working in enhancement to QEMU to let it natively handle LUKS format
volumes. At which point we'll stop using dm-crypt for for anything and
do it all in QEMU.

Nova currently decides whether it wants to use the in-kernel network
client, or an in-QEMU network client for the various network backed
storage drivers. If os-brick takes over encryption setup with dm-crypt,
then it would potentially be taking the decision away from Nova about
whether to use in-kernel or in-QEMU clients, which is not desirable.
Nova must retain control over which configuration approach is best
for the hypervisor it is using.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][nova]Move encryptors to os-brick

[Daniel P. Berrange]

On Fri, Nov 20, 2015 at 02:45:15PM +0200, Duncan Thomas wrote:
> Brick does not have to take over the decisions in order to be a useful
> repository for the code. The motivation for this work is to avoid having
> the dm setup code copied wholesale into cinder, where it becomes difficult
> to keep in sync with the code in nova.
> 
> Cinder needs a copy of this code since it is on the data path for certain
> operations (create from image, copy to image, backup/restore, migrate).

A core goal of using volume encryption in Nova to provide protection for
tenant data, from a malicious storage service. ie if the decryption key
is only ever used by Nova on the compute node, then cinder only ever sees
ciphertext, never plaintext.  Thus if cinder is compromised, then it can
not compromise any data stored in any encrypted volumes.

If cinder is looking to get access to the dm-seutp code, this seems to
imply that cinder will be getting access to the plaintext data, which
feels to me like it de-values the volume encryption feature somewhat.

I'm fuzzy on the details of just what code paths cinder needs to be
able to convert from plaintext to ciphertext or vica-verca, but in
general I think it is desirable if we can avoid any such operation
in cinder, and keep it so that only Nova compute nodes ever see the
decrypted data.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][nova]Move encryptors to os-brick

[Daniel P. Berrange]

On Fri, Nov 20, 2015 at 02:44:17PM -0500, Ben Swartzlander wrote:
> On 11/20/2015 01:19 PM, Daniel P. Berrange wrote:
> >On Fri, Nov 20, 2015 at 02:45:15PM +0200, Duncan Thomas wrote:
> >>Brick does not have to take over the decisions in order to be a useful
> >>repository for the code. The motivation for this work is to avoid having
> >>the dm setup code copied wholesale into cinder, where it becomes difficult
> >>to keep in sync with the code in nova.
> >>
> >>Cinder needs a copy of this code since it is on the data path for certain
> >>operations (create from image, copy to image, backup/restore, migrate).
> >
> >A core goal of using volume encryption in Nova to provide protection for
> >tenant data, from a malicious storage service. ie if the decryption key
> >is only ever used by Nova on the compute node, then cinder only ever sees
> >ciphertext, never plaintext.  Thus if cinder is compromised, then it can
> >not compromise any data stored in any encrypted volumes.
> 
> There is a difference between the cinder service and the storage controller
> (or software system) that cinder manages. You can give the decryption keys
> to the cinder service without allowing the storage controller to see any
> plaintext.
> 
> As Walt says in the relevant patch [1], expecting cinder to do data
> management without ever performing I/O is unrealistic. The scenario where
> the compute admin doesn't trust the storage admin is understandable
> (although less important than other potential types of attacks IMO) but the
> scenario where the guy managing nova doesn't trust the guy managing cinder
> makes no sense at all.

So you are implicitly saying here that the cinder admin is different from
the storage admin. While that certainly may often be true, I strugle to
categorically say it is always going to be true.

Furthermore it is not only about the trust of the cinder administrator,
but rather trust of the integrity of the cinder service. OpenStack has
a great many components that are open to attack, and it is prudent to
design the system such that successfull security attacks are confined
to as large a degree as possible. From this POV I think it is entirely
reasonable & indeed sensible for Nova to have minimal trust of Cinder
as a whole when it comes to tenant data security.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][nova]Move encryptors to os-brick

[Daniel P. Berrange]

On Fri, Nov 20, 2015 at 11:34:29AM -0800, Walter A. Boring IV wrote:
> On 11/20/2015 10:19 AM, Daniel P. Berrange wrote:
> >On Fri, Nov 20, 2015 at 02:45:15PM +0200, Duncan Thomas wrote:
> >>Brick does not have to take over the decisions in order to be a useful
> >>repository for the code. The motivation for this work is to avoid having
> >>the dm setup code copied wholesale into cinder, where it becomes difficult
> >>to keep in sync with the code in nova.
> >>
> >>Cinder needs a copy of this code since it is on the data path for certain
> >>operations (create from image, copy to image, backup/restore, migrate).
> >A core goal of using volume encryption in Nova to provide protection for
> >tenant data, from a malicious storage service. ie if the decryption key
> >is only ever used by Nova on the compute node, then cinder only ever sees
> >ciphertext, never plaintext.  Thus if cinder is compromised, then it can
> >not compromise any data stored in any encrypted volumes.
> >
> >If cinder is looking to get access to the dm-seutp code, this seems to
> >imply that cinder will be getting access to the plaintext data, which
> >feels to me like it de-values the volume encryption feature somewhat.
> >
> >I'm fuzzy on the details of just what code paths cinder needs to be
> >able to convert from plaintext to ciphertext or vica-verca, but in
> >general I think it is desirable if we can avoid any such operation
> >in cinder, and keep it so that only Nova compute nodes ever see the
> >decrypted data.
> Being able to limit the number of points where an encrypted volume can be
> used unencrypted
> is obviously a good goal.
> Unfortunately, it's entirely unrealistic to expect Cinder to never be able
> to have access that access.
>
> Cinder currently needs access to write data to volumes that are encrypted
> for several operations.
> 
> 1) copy volume to image

If a volume is encrypted and it is being copied to an image, IMHO we
should not aim to decrypt it. We should copy the data as is and mark
the image as encrypted in glance, and then use it as is next time the
image is needed.

FYI, Nova is already aiming to consider both the glance data storage
and the glance service as a whole, as untrustworthy. The first step
in this is using cryptographic signatures to detect unauthorized
image data modification by a compromised glance. Encryption will be
a later step in the process.

> 2) copy image to volume

This is semi-plausible as a place where Cinder needs to go from
unencrypted image data to encrypted volume data, when a user is
creating a volume from an image ahead of time, distinct from any
VM boot attempt. In such a case it is desirable that Cinder not
be able to request any existing volume keys from the key server,
merely have the ability to upload new keys and throw away its
local copy thereafter.

> 3) backup

Cinder should really not try to decrypt volumes when backing them
up. If it conversely wants to encrypt volumes during backup, it
can do so with separate backup keys, distinct from those used for
primary volume encryption for use at runtime.


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Migration progress

[Daniel P. Berrange]

On Mon, Nov 23, 2015 at 08:36:32AM +, Paul Carlton wrote:
> John
> 
> At the live migration sub team meeting I undertook to look at the issue
> of progress reporting.
> 
> The use cases I'm envisaging are...
> 
> As a user I want to know how much longer my instance will be migrating
> for.
> 
> As an operator I want to identify any migration that are making slow
>  progress so I can expedite their progress or abort them.
> 
> The current implementation reports on the instance's migration with
> respect to memory transfer, using the total memory and memory remaining
> fields from libvirt to report the percentage of memory still to be
> transferred.  Due to the instance writing to pages already transferred
> this percentage can go up as well as down.  Daniel has done a good job
> of generating regular log records to report progress and highlight lack
> of progress but from the API all a user/operator can see is the current
> percentage complete.  By observing this periodically they can identify
> instance migrations that are struggling to migrate memory pages fast
> enough to keep pace with the instance's memory updates.
> 
> The problem is that at present we have only one field, the instance
> progress, to record progress.  With a live migration there are measures
> of progress, how much of the ephemeral disks (not needed for shared
> disk setups) have been copied and how much of the memory has been
> copied. Both can go up and down as the instance writes to pages already
> copied causing those pages to need to be copied again.  As Daniel says
> in his comments in the code, the disk size could dwarf the memory so
> reporting both in single percentage number is problematic.
> 
> We could add an additional progress item to the instance object, i.e.
> disk progress and memory progress but that seems odd to have an
> additional progress field only for this operation so this is probably
> a non starter!
> 
> For operations staff with access to log files we could report disk
> progress as well as memory in the log file, however that does not
> address the needs of users and whilst log files are the right place for
> support staff to look when investigating issues operational tooling
> is much better served by notification messages.
> 
> Thus I'd recommend generating periodic notifications during a migration
> to report both memory and disk progress would be useful?  Cloud
> operators are likely to manage their instance migration activity using
> some orchestration tooling which could consume these notifications and
> deduce what challenges the instance migration is encountering and thus
> determine how to address any issues.
> 
> The use cases are only partially addressed by the current
> implementation, they can repeatedly get the server details and look at
> the progress percentage to see how quickly (or even if) it is
> increasing and determine how long the instance is likely to be
> migrating for.  However for an instance that has a large disk and/or
> is doing a high rate of disk i/o they may see the percentage complete
> (i.e. memory) repeatedly showing 90%+ but the instance migration does
> not complete.
> 
> The nova spec https://review.openstack.org/#/c/248472/ suggests making
> detailed information available via the os-migrations object.  This is
> not a bad idea but I have some issues with the implementation that I
> will share on that spec.

As I mentioned in the spec, I won't support exposing anything other
than disk total + remaining via the API. All the other stats are
low level QEMU specific implementation details that I feel the public
API users have no business knowing about.

In general I think we need to be wary of exposing lots of info + knobs
via the API, as that direction essentially ends up forcing the problem
onto client application. The focus should really be on ensuring that
Nova consumes all these stats exposed by QEMU and makes decisions
itself based on that.

At most an external application should have information on the data
transfer progress. I'm not even convinced that applications should
need to be able to figure out if a live migration is stuck. I generally
think that any scenario in which a live migration can get stuck is a
bug in Nova's management of the migration process. IOW, the focus of
our efforts should be on ensuring Nova does the right thing to guarantee
that live migration will never get stuck. At which point an Nova client
user / application should really only care about the overall progress
of a live migration.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage 

Re: [openstack-dev] [cinder][glance]Upload encrypted volumes to images

[Daniel P. Berrange]

On Mon, Nov 23, 2015 at 03:45:55AM +, Li, Xiaoyan wrote:
> Hi all,
> More help about volume encryption is needed. 
> 
> About uploading encrypted volumes to image, there are three options:
> 1. Glance only keeps non-encrypted images. So when uploading encrypted
> volumes to image, cinder de-crypts the data and upload.

This may be desirable in some cases, but for people wanting to provide
end to end encryption of all tenant data, unencrypting volumes when
converting them to images to store is glance is really the last thing
we want to do. Once tenant data is encrypted, the goal should be to
never decrypt it again except when booting an instance with the volume
or image.

> 2. Glance maintain encrypted images. Cinder just upload the encrypted
> data to image.

That is highly desirable as an option, since it allows glance to remain an
relatively untrusted component. The image signature work will soon allow
Nova to consider glance as untrusted, by allowing Nova to verify that Glance
has not tampered with the data that was provided by user, nor tried to serve
Nova data from a different user.  Following this lead, I think the ability
to prevent Glance seeing any plaintext data from the image is an obvious
beneficial step forwards.

> 3. Just prevent the function to upload encrypted volumes to images.

That's obviously fairly limiting.

> Option 1 No changes needed in Glance. But it may be not safe. As we
> decrypt the data, and upload it to images.

s/may be not safe/is not safe/.

> Option 2 This imports encryption to Glance which needs to manage the
> encryption metadata.

Glance doesn't need to do all that much besides recording a few
bits of metadata, so that doesn't seem unreasonable todo.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][glance]Upload encrypted volumes to images

[Daniel P. Berrange]

On Mon, Nov 23, 2015 at 07:05:05AM +0100, Philipp Marek wrote:
> > About uploading encrypted volumes to image, there are three options:
> > 1. Glance only keeps non-encrypted images. So when uploading encrypted 
> >volumes to image, cinder de-crypts the data and upload.
> > 2. Glance maintain encrypted images. Cinder just upload the encrypted 
> >data to image. 
> > 3. Just prevent the function to upload encrypted volumes to images.
> >
> > Option 1 No changes needed in Glance. But it may be not safe. As we decrypt 
> > the data, and upload it to images. 
> > Option 2 This imports encryption to Glance which needs to manage the 
> > encryption metadata.
> > 
> > Please add more if you have other suggestions. How do you think which one 
> > is preferred.
> Well, IMO only option 1 is useful.
> 
> Option 2 means that the original volume, the image, and all derived volumes 
> will share the same key, right?

That depends on how you implement it really. If you directly upload the
encrypted volume as-is, and then directly boot later VMs of the same
image, as-is they'll obviously share the same key. It is possible though
for cinder to re-encrypt the volume with a different key before uploading
it, or more likely for Nova to re-encrypt the image with a different key
after downloading it to boot an instance.

> That's not good. (Originally: "unacceptable")

If the images and all volumes are all owned by a single tenant user it
is not a big deal if they have the same key. Depending on what threats
you are protecting against, it may be more desirable than having the
data stored unencrypted in glance.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for live migration

[Daniel P. Berrange]

On Thu, Nov 26, 2015 at 11:55:31PM +0800, 少合冯 wrote:
> Hi all,
> We want to support xbzrle compress for live migration.
> 
> Now there are 3 options,
> 1. add the enable flag in nova.conf.
> such as a dedicated 'live_migration_compression=on|off" parameter in
> nova.conf.
> And nova simply enable it.
> seems not good.

Just having a live_migration_compression=on|off parameter that
unconditionally turns it on for all VMs is not really a solution
on its own, as it leaves out the problem of compression cache
memory size, which is at the root of the design problem.

Without sensible choice of the cache size, the compression is
either useless (to small and it won't get a useful number cache
hits and so won't save any data transfer bandwidth) or it is
hugely wasteful of resources (to large and you're just sucking
host RAM for no benefit). QEMU migration code maintainers
guidelines are that the cache size should be approximately
equal to the guest RAM working set. IOW for a 4 GB guest
you potentially need a 4 GB cache for migration, so we're
doubling the memory usage of a guest, without the schedular
being any the wiser, which will inevitably cause the host
to die in out of memory at some point.


> 2.  add a parameters in live migration API.
> 
> A new array compress will be added as optional, the json-schema as below::
> 
>   {
> 'type': 'object',
> 'properties': {
>   'os-migrateLive': {
> 'type': 'object',
> 'properties': {
>   'block_migration': parameter_types.boolean,
>   'disk_over_commit': parameter_types.boolean,
>   'compress': {
> 'type': 'array',
> 'items': ["xbzrle"],
>   },
>   'host': host
> },
> 'additionalProperties': False,
>   },
> },
> 'required': ['os-migrateLive'],
> 'additionalProperties': False,
>   }

I really don't think we want to expose this kind of hypervisor
specific detail in the live migration API of Nova. It just leaks
too many low level details. It still leaves the problem of deciding
the compression cache size unsolved and likewise the problem of the
schedular knowing about the memory usage for this cache in order to
avoid OOM

> 3.  dynamically choose when to activate xbzrle compress for live migration.
>  This is the best.
>  xbzrle really wants to be used if the network is not able to keep up
> with the dirtying rate of the guest RAM.
>  But how do I check the coming migration fit this situation？

FWIW, if we decide we want compression support in Nova, I think that
having the Nova libvirt driver dynamically decide when to use it is
the only viable approach. Unfortunately the way the QEMU support
is implemented makes it very hard to use, as QEMU forces you to decide
to use it upfront, at a time when you don't have any useful information
on which to make the decision :-(  To be useful IMHO, we really need
the ability to turn on compression on the fly for an existing active
migration process. ie, we'd start migration off and let it run and
only enable compression if we encounter problems with completion.
Sadly we can't do this with QEMU as it stands today :-(

Oh and of course we still need to address the issue of RAM usage and
communicating that need with the scheduler in order to avoid OOM
scenarios due to large compression cache.

I tend to feel that the QEMU compression code is currently broken by
design and needs rework in QEMU before it can be pratically used in
an autonomous fashion :-(

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for live migration

[Daniel P. Berrange]

On Thu, Nov 26, 2015 at 05:49:50PM +, Paul Carlton wrote:
> Seems to me the prevailing view is that we should get live migration to
> figure out the best setting for
> itself where possible.  There was discussion of being able have a default
> policy setting that will allow
> the operator to define balance between speed of migration and impact on the
> instance.  This could be
> a global default for the cloud with overriding defaults per aggregate,
> image, tenant and instance as
> well as the ability to vary the setting during the migration operation.
> 
> Seems to me that items like compression should be set in configuration files
> based on what works best
> given the cloud operator's environment?

Merely turning on use of compression is the "easy" bit - there needs to be
a way to deal with compression cache size allocation, which needs to have
some smarts in Nova, as there's no usable "one size fits all" value for
the compression cache size. If we did want to hardcode a compression cache
size, you'd have to pick set it as a scaling factor against the guest RAM
size. This is going to be very heavy on memory usage, so there needs careful
design work to solve the problem of migration compression triggering host
OOM scenarios, particularly since we can have multiple concurrent
migrations.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for live migration

[Daniel P. Berrange]

On Thu, Nov 26, 2015 at 05:39:04PM +, Daniel P. Berrange wrote:
> On Thu, Nov 26, 2015 at 11:55:31PM +0800, 少合冯 wrote:
> > 3.  dynamically choose when to activate xbzrle compress for live migration.
> >  This is the best.
> >  xbzrle really wants to be used if the network is not able to keep up
> > with the dirtying rate of the guest RAM.
> >  But how do I check the coming migration fit this situation？
> 
> FWIW, if we decide we want compression support in Nova, I think that
> having the Nova libvirt driver dynamically decide when to use it is
> the only viable approach. Unfortunately the way the QEMU support
> is implemented makes it very hard to use, as QEMU forces you to decide
> to use it upfront, at a time when you don't have any useful information
> on which to make the decision :-(  To be useful IMHO, we really need
> the ability to turn on compression on the fly for an existing active
> migration process. ie, we'd start migration off and let it run and
> only enable compression if we encounter problems with completion.
> Sadly we can't do this with QEMU as it stands today :-(
> 
> Oh and of course we still need to address the issue of RAM usage and
> communicating that need with the scheduler in order to avoid OOM
> scenarios due to large compression cache.
> 
> I tend to feel that the QEMU compression code is currently broken by
> design and needs rework in QEMU before it can be pratically used in
> an autonomous fashion :-(

Actually thinking about it, there's not really any significant
difference between Option 1 and Option 3. In both cases we want
a nova.conf setting live_migration_compression=on|off to control
whether we want to *permit* use  of compression.

The only real difference between 1 & 3 is whether migration has
compression enabled always, or whether we turn it on part way
though migration.

So although option 3 is our desired approach (which we can't
actually implement due to QEMU limitations), option 1 could
be made fairly similar if we start off with a very small
compression cache size which would have the effect of more or
less disabling compression initially.

We already have logic in the code for dynamically increasing
the max downtime value, which we could mirror here

eg something like

 live_migration_compression=on|off

  - Whether to enable use of compression

 live_migration_compression_cache_ratio=0.8

  - The maximum size of the compression cache relative to
the guest RAM size. Must be less than 1.0

 live_migration_compression_cache_steps=10

  - The number of steps to take to get from initial cache
size to the maximum cache size

 live_migration_compression_cache_delay=75

  - The time delay in seconds between increases in cache
size


In the same way that we do with migration downtime, instead of
increasing cache size linearly, we'd increase it in ever larger
steps until we hit the maximum. So we'd start off fairly small
a few MB, and monitoring the cache hit rates, we'd increase it
periodically.  If the number of steps configured and time delay
between steps are reasonably large, that would have the effect
that most migrations would have a fairly small cache and would
complete without needing much compression overhead.

Doing this though, we still need a solution to the host OOM scenario
problem. We can't simply check free RAM at start of migration and
see if there's enough to spare for compression cache, as the schedular
can spawn a new guest on the compute host at any time, pushing us into
OOM. We really need some way to indicate that there is a (potentially
very large) extra RAM overhead for the guest during migration.

ie if live_migration_compression_cache_ratio is 0.8 and we have a
4 GB guest, we need to make sure the schedular knows that we are
potentially going to be using 7.2 GB of memory during migration

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for live migration

[Daniel P. Berrange]

On Fri, Nov 27, 2015 at 07:37:50PM +0800, 少合冯 wrote:
> 2015-11-27 2:19 GMT+08:00 Daniel P. Berrange :
> 
> > On Thu, Nov 26, 2015 at 05:39:04PM +0000, Daniel P. Berrange wrote:
> > > On Thu, Nov 26, 2015 at 11:55:31PM +0800, 少合冯 wrote:
> > > > 3.  dynamically choose when to activate xbzrle compress for live
> > migration.
> > > >  This is the best.
> > > >  xbzrle really wants to be used if the network is not able to keep
> > up
> > > > with the dirtying rate of the guest RAM.
> > > >  But how do I check the coming migration fit this situation？
> > >
> > > FWIW, if we decide we want compression support in Nova, I think that
> > > having the Nova libvirt driver dynamically decide when to use it is
> > > the only viable approach. Unfortunately the way the QEMU support
> > > is implemented makes it very hard to use, as QEMU forces you to decide
> > > to use it upfront, at a time when you don't have any useful information
> > > on which to make the decision :-(  To be useful IMHO, we really need
> > > the ability to turn on compression on the fly for an existing active
> > > migration process. ie, we'd start migration off and let it run and
> > > only enable compression if we encounter problems with completion.
> > > Sadly we can't do this with QEMU as it stands today :-(
> > >
> >
> [Shaohe Feng]
> Add more guys working on kernel/hypervisor in our loop.
> Wonder whether there will be any good solutions to improve it in QEMU in
> future.
> 
> 
> > > Oh and of course we still need to address the issue of RAM usage and
> > > communicating that need with the scheduler in order to avoid OOM
> > > scenarios due to large compression cache.
> > >
> > > I tend to feel that the QEMU compression code is currently broken by
> > > design and needs rework in QEMU before it can be pratically used in
> > > an autonomous fashion :-(
> >
> > Actually thinking about it, there's not really any significant
> > difference between Option 1 and Option 3. In both cases we want
> > a nova.conf setting live_migration_compression=on|off to control
> > whether we want to *permit* use  of compression.
> >
> > The only real difference between 1 & 3 is whether migration has
> > compression enabled always, or whether we turn it on part way
> > though migration.
> >
> > So although option 3 is our desired approach (which we can't
> > actually implement due to QEMU limitations), option 1 could
> > be made fairly similar if we start off with a very small
> > compression cache size which would have the effect of more or
> > less disabling compression initially.
> >
> > We already have logic in the code for dynamically increasing
> > the max downtime value, which we could mirror here
> >
> > eg something like
> >
> >  live_migration_compression=on|off
> >
> >   - Whether to enable use of compression
> >
> >  live_migration_compression_cache_ratio=0.8
> >
> >   - The maximum size of the compression cache relative to
> > the guest RAM size. Must be less than 1.0
> >
> >  live_migration_compression_cache_steps=10
> >
> >   - The number of steps to take to get from initial cache
> > size to the maximum cache size
> >
> >  live_migration_compression_cache_delay=75
> >
> >   - The time delay in seconds between increases in cache
> > size
> >
> >
> > In the same way that we do with migration downtime, instead of
> > increasing cache size linearly, we'd increase it in ever larger
> > steps until we hit the maximum. So we'd start off fairly small
> > a few MB, and monitoring the cache hit rates, we'd increase it
> > periodically.  If the number of steps configured and time delay
> > between steps are reasonably large, that would have the effect
> > that most migrations would have a fairly small cache and would
> > complete without needing much compression overhead.
> >
> > Doing this though, we still need a solution to the host OOM scenario
> > problem. We can't simply check free RAM at start of migration and
> > see if there's enough to spare for compression cache, as the schedular
> > can spawn a new guest on the compute host at any time, pushing us into
> > OOM. We really need some way to indicate that there is a (potentially
> > very large) extra RAM overhead for the guest during migration.
> >
> > ie if live_migration_compression_cache_ratio is 0.8 

Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for live migration

[Daniel P. Berrange]

On Fri, Nov 27, 2015 at 12:17:06PM +, Koniszewski, Pawel wrote:
> > -Original Message-
> > > > Doing this though, we still need a solution to the host OOM scenario
> > > > problem. We can't simply check free RAM at start of migration and
> > > > see if there's enough to spare for compression cache, as the
> > > > schedular can spawn a new guest on the compute host at any time,
> > > > pushing us into OOM. We really need some way to indicate that there
> > > > is a (potentially very large) extra RAM overhead for the guest during
> > migration.
> 
> What about CPU? We might end up with live migration that degrades
> performance of other VMs on source and/or destination node. AFAIK
> CPUs are heavily oversubscribed in many cases and this does not help.
> I'm not sure that this thing fits into Nova as it requires resource
> monitoring.

Nova already has the ability to set CPU usage tuning rules against
each VM. Since the CPU overhead is attributed to the QEMU process,
these existing tuning rules will apply. So there would only be
impact on other VMs, if you do not have any CPU tuning rules set
in Nova.


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for live migration

[Daniel P. Berrange]

On Fri, Nov 27, 2015 at 01:01:15PM +, Koniszewski, Pawel wrote:
> > -Original Message-
> > From: Daniel P. Berrange [mailto:berra...@redhat.com]
> > Sent: Friday, November 27, 2015 1:24 PM
> > To: Koniszewski, Pawel
> > Cc: OpenStack Development Mailing List (not for usage questions); ???; Feng,
> > Shaohe; Xiao, Guangrong; Ding, Jian-feng; Dong, Eddie; Wang, Yong Y; Jin,
> > Yuntong
> > Subject: Re: [openstack-dev] [nova] [RFC] how to enable xbzrle compress for
> > live migration
> >
> > On Fri, Nov 27, 2015 at 12:17:06PM +, Koniszewski, Pawel wrote:
> > > > -Original Message-
> > > > > > Doing this though, we still need a solution to the host OOM
> > > > > > scenario problem. We can't simply check free RAM at start of
> > > > > > migration and see if there's enough to spare for compression
> > > > > > cache, as the schedular can spawn a new guest on the compute
> > > > > > host at any time, pushing us into OOM. We really need some way
> > > > > > to indicate that there is a (potentially very large) extra RAM
> > > > > > overhead for the guest during
> > > > migration.
> > >
> > > What about CPU? We might end up with live migration that degrades
> > > performance of other VMs on source and/or destination node. AFAIK CPUs
> > > are heavily oversubscribed in many cases and this does not help.
> > > I'm not sure that this thing fits into Nova as it requires resource
> > > monitoring.
> >
> > Nova already has the ability to set CPU usage tuning rules against each VM.
> > Since the CPU overhead is attributed to the QEMU process, these existing
> > tuning rules will apply. So there would only be impact on other VMs, if you
> > do
> > not have any CPU tuning rules set in Nova.
> 
> Not sure I understand it correctly, I assume that you are talking about CPU
> pinning. Does it mean that compression/decompression runs as part of VM
> threads?
> 
> If not then, well, it will require all VMs to be pinned on both hosts, source
> and destination (and in the whole cluster because of static configuration...).
> Also what about operating system performance? Will QEMU distinct OS processes
> somehow and won't affect them?

The compression runs in the migration thread of QEMU. This is not a vCPU
thread, but one of the QEMU emulator threads. So CPU usage policy set
against the QEMU emulator threads applies to the compression CPU overhead.

> Also, nova can reserve some memory for the host. Will QEMU also respect it?

No, its not QEMU's job to respect that. If you want to reserve resources
for only the host OS, then you need to setup suitable cgroup partitions
to separate VM from non-VM processes. The Nova reserved memory setting
is merely a hint to the schedular - it has no functional effect on its
own.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Testing concerns around boot from UEFI spec

[Daniel P. Berrange]

On Fri, Dec 04, 2015 at 07:43:41AM -0500, Sean Dague wrote:
> Can someone explain the licensing issue here? The Fedora comments make
> this sound like this is something that's not likely to end up in distros.

The EDK codebase contains a FAT driver which has a license that forbids
reusing the code outside of the EDK project.

[quote]
Additional terms: In addition to the forgoing, redistribution and use
of the code is conditioned upon the FAT 32 File System Driver and all
derivative works thereof being used for and designed only to read
and/or write to a file system that is directly managed by Intel's
Extensible Firmware Initiative (EFI) Specification v. 1.0 and later
and/or the Unified Extensible Firmware Interface (UEFI) Forum's UEFI
Specifications v.2.0 and later (together the "UEFI Specifications");
only as necessary to emulate an implementation of the UEFI Specifications;
and to create firmware, applications, utilities and/or drivers.
[/quote]

So while the code is open source, it is under a non-free license,
hence Fedora will not ship it. For RHEL we're reluctantly choosing
to ship it as an exception to our normal policy, since its the only
immediate way to make UEFI support available on x86 & aarch64

So I don't think the license is a reason to refuse to allow the UEFI
feature into Nova though, nor should it prevent us using the current
EDK bios in CI for testing purposes. It is really just an issue for
distros which only want 100% free software.

Unless the license on the existing code gets resolved, some Red Hat
maintainers have a plan to replace the existing FAT driver with an
alternative impl likely under GPL. At that time, it'll be acceptable
for inclusion in Fedora.

> That seems weird enough that I'd rather push back on our Platinum Board
> member to fix the licensing before we let this in. Especially as this
> feature is being drive by Intel.

As copyright holder, Intel could choose to change the license of their
code to make it free software avoiding all the problems. None the less,
as above, I don't think this is a blocker for inclusion of the feature
in Nova, nor our testing of it.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Thoughts on deprecating the legacy bdm v1 API support

[Daniel P. Berrange]

On Tue, May 17, 2016 at 09:48:02AM -0500, Matt Riedemann wrote:
> In the live migration meeting today mdbooth and I were chatting about how
> hard it is to follow the various BDM code through nova, because you have the
> three block_device modules:
> 
> * nova.block_device - dict that does some translation magic
> * nova.objects.block_device - contains the BDM(List) objects for RPC and DB
> access
> * nova.virt.block_device - dict that wraps a BDM object, used for attaching
> volumes to instances, updates the BDM.connection_info field in the DB via
> the wrapper on the BDM object. This module also has translation logic in it.
> 
> The BDM v1 extension translates that type of request to the BDM v2 model
> before it gets to server create, and then passed down to the
> nova.compute.api. But there is still a lot of legacy bdm v1 translation
> logic spread through the code.
> 
> So I'd like to propose that we deprecate the v1 BDM API in the same vein
> that we're deprecating other untested things, like agent-builds, cloudpipe,
> certificates, and the proxy APIs. We can't remove the code, but we can
> signal to users to not use the API and eventually when we raise the minimum
> required microversion >= the deprecation, we can drop that code. Since
> that's a long ways off, the earlier we start a deprecation clock on this the
> better - if we're going to do it.

Given that actual deletion of the code is a long way off regardless, can we
at least figure out how to isolate the BDM v1 support so that it only exists
in the very top most API entrypoint, and gets immediately converted to v2.
That way 95% of nova would not have to know / care about it, even if we don't
finally drop v1 for 10 more years.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova]Libvirt error code for failing volume in nova

[Daniel P. Berrange]

On Wed, May 18, 2016 at 07:57:17PM +, Radhakrishnan, Siva wrote:
> Hi All!
> Currently I am working on this bug 
> https://bugs.launchpad.net/nova/+bug/1168011 which says we have to change  
> error message displayed  when attaching a volume fails. Currently it catches 
> all operation errors that libvirt can raise and assume that all of them are 
> the source of device being busy. You can find the source of this code here 
> https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1160.
>  I have few questions on this bug
>  
> 1.what kind of error message and other required info should we include in 
> the exception to make it look more  generalized instead of the current 
> one ?
>  
> 2.Should we raise separate exception for "Device is Busy" or a single 
> general exception would work fine ?
>  
> 3.If we need separate exception for device being busy what would
> be the equivalent libvirt error code  for that

There is not any specific libvirt error code used for this situation that
you can detect, which is why the code is catching the general libvirt
error. 

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] [nova] live migration, libvirt 1.3, and the gate

[Daniel P. Berrange]

On Tue, May 24, 2016 at 01:59:17PM -0400, Sean Dague wrote:
> The team working on live migration testing started with an experimental
> job on Ubuntu 16.04 to try to be using the latest and greatest libvirt +
> qemu under the assumption that a set of issues we were seeing are
> solved. The short answer is, it doesn't look like this is going to work.
> 
> We run tests on a bunch of different clouds. Those clouds expose
> different cpu flags to us. These are not standard things that map to
> "Haswell". It means live migration in the multinode cases can hit cpus
> with different flags. So we found the requirement was to come up with a
> least common denominator of cpu flags, which we call gate64, and push
> that into the libvirt cpu_map.xml in devstack, and set whenever we are
> in a multinode scenario.
> (https://github.com/openstack-dev/devstack/blob/master/tools/cpu_map_update.py)
>  Not ideal, but with libvirt 1.2.2 it works fine.
> 
> It turns out it works fine because libvirt *actually* seems to take the
> data from cpu_map.xml and do a translation to what it believes qemu will
> understand. On these systems apparently this turns into "-cpu
> Opteron_G1,-pse36"
> (http://logs.openstack.org/29/42529/24/check/gate-tempest-dsvm-multinode-full/5f504c5/logs/libvirt/qemu/instance-000b.txt.gz)
> 
> At some point between libvirt 1.2.2 and 1.3.1, this changed. Now libvirt
> seems to be passing our cpu_model directly to qemu, and assumes that as
> a user you will be responsible for writing all the  stanzas to
> add/remove yourself. When libvirt sends 'gate64' to qemu, this explodes,
> as qemu has no idea what we are talking about.
> http://logs.openstack.org/34/319934/2/experimental/gate-tempest-dsvm-multinode-live-migration/b87d689/logs/screen-n-cpu.txt.gz#_2016-05-24_15_59_12_531
> 
> Unlike libvirt, which has a text file (xml) that configures the cpus
> that could exist in the world, qemu builds this in statically at compile
> time:
> http://git.qemu.org/?p=qemu.git;a=blob;f=target-i386/cpu.c;h=895a386d3b7a94e363ca1bb98821d3251e70c0e0;hb=HEAD#l694
> 
> 
> So, the existing cpu_map.xml workaround for our testing situation will
> no longer work.
> 
> So, we have a number of open questions:
> 
> * Have our cloud providers standardized enough that we might get away
> without this custom cpu model? (Have some of them done it and only use
> those for multinode?)
> * Is there any way to get this feature back in libvirt to do the cpu
> computation?
> * Would we have to build a whole nova feature around setting libvirt xml
>  to be able to test live migration in our clouds?
> * Other options?
> * Do we give up and go herd goats?

Rather than try to define our own custom CPU models, we can probably
just use one of the standard CPU models and then explicitly tell
libvirt which flags to turn off in order to get compatibility with
our cloud environments.

This is not currently possible with Nova, since our nova.conf option
only allow us to specify a bare CPU model. We would have to extend
nova.conf to allow us to specify a list of CPU features to add or
remove. Libvirt should then correctly pass these changes through
to QEMU.


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] [nova] live migration, libvirt 1.3, and the gate

[Daniel P. Berrange]

On Tue, May 31, 2016 at 08:24:03AM -0400, Sean Dague wrote:
> On 05/31/2016 05:39 AM, Daniel P. Berrange wrote:
> > On Tue, May 24, 2016 at 01:59:17PM -0400, Sean Dague wrote:
> >> The team working on live migration testing started with an experimental
> >> job on Ubuntu 16.04 to try to be using the latest and greatest libvirt +
> >> qemu under the assumption that a set of issues we were seeing are
> >> solved. The short answer is, it doesn't look like this is going to work.
> >>
> >> We run tests on a bunch of different clouds. Those clouds expose
> >> different cpu flags to us. These are not standard things that map to
> >> "Haswell". It means live migration in the multinode cases can hit cpus
> >> with different flags. So we found the requirement was to come up with a
> >> least common denominator of cpu flags, which we call gate64, and push
> >> that into the libvirt cpu_map.xml in devstack, and set whenever we are
> >> in a multinode scenario.
> >> (https://github.com/openstack-dev/devstack/blob/master/tools/cpu_map_update.py)
> >>  Not ideal, but with libvirt 1.2.2 it works fine.
> >>
> >> It turns out it works fine because libvirt *actually* seems to take the
> >> data from cpu_map.xml and do a translation to what it believes qemu will
> >> understand. On these systems apparently this turns into "-cpu
> >> Opteron_G1,-pse36"
> >> (http://logs.openstack.org/29/42529/24/check/gate-tempest-dsvm-multinode-full/5f504c5/logs/libvirt/qemu/instance-000b.txt.gz)
> >>
> >> At some point between libvirt 1.2.2 and 1.3.1, this changed. Now libvirt
> >> seems to be passing our cpu_model directly to qemu, and assumes that as
> >> a user you will be responsible for writing all the  stanzas to
> >> add/remove yourself. When libvirt sends 'gate64' to qemu, this explodes,
> >> as qemu has no idea what we are talking about.
> >> http://logs.openstack.org/34/319934/2/experimental/gate-tempest-dsvm-multinode-live-migration/b87d689/logs/screen-n-cpu.txt.gz#_2016-05-24_15_59_12_531
> >>
> >> Unlike libvirt, which has a text file (xml) that configures the cpus
> >> that could exist in the world, qemu builds this in statically at compile
> >> time:
> >> http://git.qemu.org/?p=qemu.git;a=blob;f=target-i386/cpu.c;h=895a386d3b7a94e363ca1bb98821d3251e70c0e0;hb=HEAD#l694
> >>
> >>
> >> So, the existing cpu_map.xml workaround for our testing situation will
> >> no longer work.
> >>
> >> So, we have a number of open questions:
> >>
> >> * Have our cloud providers standardized enough that we might get away
> >> without this custom cpu model? (Have some of them done it and only use
> >> those for multinode?)
> >> * Is there any way to get this feature back in libvirt to do the cpu
> >> computation?
> >> * Would we have to build a whole nova feature around setting libvirt xml
> >>  to be able to test live migration in our clouds?
> >> * Other options?
> >> * Do we give up and go herd goats?
> > 
> > Rather than try to define our own custom CPU models, we can probably
> > just use one of the standard CPU models and then explicitly tell
> > libvirt which flags to turn off in order to get compatibility with
> > our cloud environments.
> > 
> > This is not currently possible with Nova, since our nova.conf option
> > only allow us to specify a bare CPU model. We would have to extend
> > nova.conf to allow us to specify a list of CPU features to add or
> > remove. Libvirt should then correctly pass these changes through
> > to QEMU.
> 
> Yes, that's an option. Given that the libvirt team seemed to acknowledge
> this as a regression, I'd rather not build a user exposed feature for
> all of that just as a workaround for a libvirt regression.

I think that fact that we're hitting this problem in the gate though is
a sign that our users will likely hit it in their own deployments if
using virtualized hosts. I think it is more friendly for users to be
able to customize the CPU features via nova.conf, then to repeat the
hacks done for devstack with editing the libvirt cpu_map.xml file.

IOW, extending nova.conf to support this officially would be a generally
useful feature for nova, beyond your short term CI needs.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [gate] [nova] live migration, libvirt 1.3, and the gate

[Daniel P. Berrange]

On Tue, May 31, 2016 at 08:19:33AM -0400, Sean Dague wrote:
> On 05/30/2016 06:25 AM, Kashyap Chamarthy wrote:
> > On Thu, May 26, 2016 at 10:55:47AM -0400, Sean Dague wrote:
> >> On 05/26/2016 05:38 AM, Kashyap Chamarthy wrote:
> >>> On Wed, May 25, 2016 at 05:42:04PM +0200, Kashyap Chamarthy wrote:
> >>>
> >>> [...]
> >>>
>  So, in short, the central issue seems to be this: the custom 'gate64'
>  model is not being trasnalted by libvirt into a model that QEMU can
>  recognize.
> >>>
> >>> An update:
> >>>
> >>> Upstream libvirt points out that this turns to be regression, and
> >>> bisected it to commit (in libvirt Git): 1.2.9-31-g445a09b -- "qemu:
> >>> Don't compare CPU against host for TCG".
> >>>
> >>> So, I expect there's going to be fix pretty soon upstream libvirt.
> >>
> >> Which is good... I wonder how long we'll be waiting for that back in our
> >> distro packages though.
> > 
> > Yeah, until the fix lands, our current options seem to be:
> > 
> >   (a) Revert to a known good version of libvirt
> 
> Downgrading libvirt so dramatically isn't a thing we'll be able to do.
> 
> >   (b) Use nested virt (i.e. ) -- I doubt is possible
> >   on RAX environment, which is using Xen, last I know.
> 
> We turned off nested virt even where it was enabled, because it locks up
> at a non trivial rate. So not really an option.

Hmm, if the guest is using 'qemu' and not 'kvm', then there should be
no dependancy between the host CPU and guest CPU whatsoever. ie we can
present arbitrary CPU to the guest, whether the host CPU has matching
features or not.

I wonder if there is a bug in Nova where it is trying todo a host/guest
CPU compatibility check even for 'qemu' guests, when it should only do
them for 'kvm' guests.

If we can avoid the CPU compatibility check with qemu guest, then the
fact that there's a libvirt bug here should be irrelevant, and we could
avoid needing to invent a gate64 CPU model too.


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

[Daniel P. Berrange]

On Fri, Jun 03, 2016 at 12:32:17PM +, Paul Michali wrote:
> Hi!
> 
> I've been playing with Liberty code a bit and had some questions that I'm
> hoping Nova folks may be able to provide guidance on...
> 
> If I set up a flavor with hw:mem_page_size=2048, and I'm creating (Cirros)
> VMs with size 1024, will the scheduling use the minimum of the number of

1024 what units ? 1024 MB, or 1024 huge pages aka 2048 MB ?

> huge pages available and the size requested for the VM, or will it base
> scheduling only on the number of huge pages?
> 
> It seems to be doing the latter, where I had 1945 huge pages free, and
> tried to create another VM (1024) and Nova rejected the request with "no
> hosts available".

>From this I'm guessing you're meaning 1024 huge pages aka 2 GB earlier.

Anyway, when you request huge pages to be used for a flavour, the
entire guest RAM must be able to be allocated from huge pages.
ie if you have a guest with 2 GB of RAM, you must have 2 GB worth
of huge pages available. It is not possible for a VM to use
1.5 GB of huge pages and 500 MB of normal sized pages.

> Is this still the same for Mitaka?

Yep, this use of huge pages has not changed.

> Where could I look in the code to see how the scheduling is determined?

Most logic related to huge pages is in nova/virt/hardware.py

> If I use mem_page_size=large (what I originally had), should it evenly
> assign huge pages from the available NUMA nodes (there are two in my case)?
> 
> It looks like it was assigning all VMs to the same NUMA node (0) in this
> case. Is the right way to change to 2048, like I did above?

Nova will always avoid spreading your VM across 2 host NUMA nodes,
since that gives bad performance characteristics. IOW, it will always
allocate huge pages from the NUMA node that the guest will run on. If
you explicitly want your VM to spread across 2 host NUMA nodes, then
you must tell nova to create 2 *guest* NUMA nodes for the VM. Nova
will then place each guest NUMA node, on a separate host NUMA node
and allocate huge pages from node to match. This is done using
the hw:numa_nodes=2 parameter on the flavour

> Again, has this changed at all in Mitaka?

Nope. Well aside from random bug fixes.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Using image metadata to sanity check supplied authentication data at nova 'create' or 'recreate' time?

[Daniel P. Berrange]

On Tue, Jun 07, 2016 at 09:37:25AM -0400, Jim Rollenhagen wrote:
> On Tue, Jun 07, 2016 at 08:31:35AM +1000, Michael Still wrote:
> > On Tue, Jun 7, 2016 at 7:41 AM, Clif Houck  wrote:
> > 
> > > Hello all,
> > >
> > > At Rackspace we're running into an interesting problem: Consider a user
> > > who boots an instance in Nova with an image which only supports SSH
> > > public-key authentication, but the user doesn't provide a public key in
> > > the boot request. As far as I understand it, today Nova will happily
> > > boot that image and it may take the user some time to realize their
> > > mistake when they can't login to the instance.
> > >
> > 
> > What about images where the authentication information is inside the image?
> > For example, there's just a standard account baked in that everyone knows
> > about? In that case Nova doesn't need to inject anything into the instance,
> > and therefore the metadata doesn't need to supply anything.
> 
> Right, so that's a third case. How I'd see this working is maybe an
> image property called "auth_requires" that could be one of ["none",
> "ssh_key", "x509_cert", "password"]. Or maybe it could be multiple
> values that are OR'd, so for example an image could require an ssh key
> or an x509 cert. If the "auth_requires" property isn't found, default to
> "none" to maintain compatibility, I guess.

NB, even if you have an image that requires an SSH key to be provided in
order to enable login, it is sometimes valid to not provide one. Not least
during development, I'm often testing images which would ordinarily require
an SSH key, but I don't actually need the ability to login, so I don't bother
to provide one.

So if we provided this ability to tag images as needing an ssh key, and then
enforced that, we would then also need to extend the API to provide a way to
tell nova to explicitly ignore this and not bother enforcing it, despite what
the image metadata says.

I'm not particularly convinced the original problem is serious enough to
warrant building such a solution. It feels like the kind of mistake that
people would do once, and then learn their mistake thereafter. IOW the
consequences of the mistake don't seem particularly severe really.

> The bigger question here is around hitting the images API syncronously
> during a boot request, and where/how/if to cache the metadata that's
> returned so we don't have to do it so often. I don't have a good answer
> for that, though.

Nova already uses image metadata for countless things during the VM boot
request, so there's nothin new in this respect. We only query glance
once, thereafter the image metadata is cached by Nova in the DB on a per
instance basis, because we need to be isolated from later changes to the
metadata in glance after the VM boots.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

[Daniel P. Berrange]

On Thu, Jun 09, 2016 at 12:35:06PM -0600, Chris Friesen wrote:
> On 06/09/2016 05:15 AM, Paul Michali wrote:
> > 1) On the host, I was seeing 32768 huge pages, of 2MB size.
> 
> Please check the number of huge pages _per host numa node_.
> 
> > 2) I changed mem_page_size from 1024 to 2048 in the flavor, and then when 
> > VMs
> > were created, they were being evenly assigned to the two NUMA nodes. Each 
> > using
> > 1024 huge pages. At this point I could create more than half, but when there
> > were 1945 pages left, it failed to create a VM. Did it fail because the
> > mem_page_size was 2048 and the available pages were 1945, even though we 
> > were
> > only requesting 1024 pages?
> 
> I do not think that "1024" is a valid page size (at least for x86).

Correct, 4k, 2M and 1GB are valid page sizes.

> Valid mem_page_size values are determined by the host CPU.  You do not need
> a larger page size for flavors with larger memory sizes.

Though note that page sizes should be a multiple of favour mem size
unless you want to waste memory. eg if you have a flavour with 750MB
RAM, then you probably don't want to use 1GB pages as it waste 250MB

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][os-vif] Expanding vif capability for wiring trunk ports

[Daniel P. Berrange]

On Thu, Jun 09, 2016 at 05:31:13PM -0600, Carl Baldwin wrote:
> Hi,
> 
> You may or may not be aware of the vlan-aware-vms effort [1] in
> Neutron.  If not, there is a spec and a fair number of patches in
> progress for this.  Essentially, the goal is to allow a VM to connect
> to multiple Neutron networks by tagging traffic on a single port with
> VLAN tags.
> 
> This effort will have some effect on vif plugging because the datapath
> will include some changes that will effect how vif plugging is done
> today.
> 
> The design proposal for trunk ports with OVS adds a new bridge for
> each trunk port.  This bridge will demux the traffic and then connect
> to br-int with patch ports for each of the networks.  Rawlin Peters
> has some ideas for expanding the vif capability to include this
> wiring.
> 
> There is also a proposal for connecting to linux bridges by using
> kernel vlan interfaces.
> 
> This effort is pretty important to Neutron in the Newton timeframe.  I
> wanted to send this out to start rounding up the reviewers and other
> participants we need to see how we can start putting together a plan
> for nova integration of this feature (via os-vif?).

I've not taken a look at the proposal, but on the timing side of things
it is really way to late to start this email thread asking for design
input from os-vif or nova. We're way past the spec proposal deadline
for Nova in the Newton cycle, so nothing is going to happen until the
Ocata cycle no matter what Neutron want  in Newton. For os-vif our
focus right now is exclusively on getting existing functionality ported
over, and integrated into Nova in Newton. So again we're not really looking
to spend time on further os-vif design work right now.

In the Ocata cycle we'll be looking to integrate os-vif into Neutron to
let it directly serialize VIF objects and send them over to Nova, instead
of using the ad-hoc port-binding dicts.  From the Nova side, we're not
likely to want to support any new functionality that affects port-binding
data until after Neutron is converted to os-vif. So Ocata at the earliest,
but probably more like P, unless the Neutron conversion to os-vif gets
completed unexpectedly quickly.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Invitation to join Hangzhou Bug Smash

[Daniel P. Berrange]

On Mon, Jun 13, 2016 at 08:06:50AM +, Wang, Shane wrote:
> Hi, OpenStackers,
> 
> As you know, Huawei, Intel and CESI are hosting the 4th China OpenStack Bug 
> Smash at Hangzhou, China.
> The 1st China Bug Smash was at Shanghai, the 2nd was at Xi'an, and the 3rd 
> was at Chengdu.
> 
> We are constructing the etherpad page for registration, and the date will
> be around July 11 (probably July 6 - 8, but to be determined very soon).

The newton-2 milestone release date is July 15th, so you certainly *don't*
want the event during that week. IOW, the 8th July is the latest you should
schedule it - don't let it slip into the next week starting July 11th, as
during the week of the n-2 milestone focus of the teams will be almost
exclusively on prep for that release, to the detriment of any bug smash
event.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][os-vif] Expanding vif capability for wiring trunk ports

[Daniel P. Berrange]

On Mon, Jun 13, 2016 at 02:08:30PM +0200, Armando M. wrote:
> On 13 June 2016 at 10:35, Daniel P. Berrange  wrote:
> 
> > On Thu, Jun 09, 2016 at 05:31:13PM -0600, Carl Baldwin wrote:
> > > Hi,
> > >
> > > You may or may not be aware of the vlan-aware-vms effort [1] in
> > > Neutron.  If not, there is a spec and a fair number of patches in
> > > progress for this.  Essentially, the goal is to allow a VM to connect
> > > to multiple Neutron networks by tagging traffic on a single port with
> > > VLAN tags.
> > >
> > > This effort will have some effect on vif plugging because the datapath
> > > will include some changes that will effect how vif plugging is done
> > > today.
> > >
> > > The design proposal for trunk ports with OVS adds a new bridge for
> > > each trunk port.  This bridge will demux the traffic and then connect
> > > to br-int with patch ports for each of the networks.  Rawlin Peters
> > > has some ideas for expanding the vif capability to include this
> > > wiring.
> > >
> > > There is also a proposal for connecting to linux bridges by using
> > > kernel vlan interfaces.
> > >
> > > This effort is pretty important to Neutron in the Newton timeframe.  I
> > > wanted to send this out to start rounding up the reviewers and other
> > > participants we need to see how we can start putting together a plan
> > > for nova integration of this feature (via os-vif?).
> >
> > I've not taken a look at the proposal, but on the timing side of things
> > it is really way to late to start this email thread asking for design
> > input from os-vif or nova. We're way past the spec proposal deadline
> > for Nova in the Newton cycle, so nothing is going to happen until the
> > Ocata cycle no matter what Neutron want  in Newton.
> 
> 
> For sake of clarity, does this mean that the management of the os-vif
> project matches exactly Nova's, e.g. same deadlines and processes apply,
> even though the core team and its release model are different from Nova's?
> I may have erroneously implied that it wasn't, also from past talks I had
> with johnthetubaguy.

No, we don't intend to force ourselves to only release at milestones
like nova does. We'll release the os-vif library whenever there is new
functionality in its code that we need to make available to nova/neutron.
This could be as frequently as once every few weeks.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][os-vif] Expanding vif capability for wiring trunk ports

[Daniel P. Berrange]

On Mon, Jun 13, 2016 at 07:39:29AM -0400, Assaf Muller wrote:
> On Mon, Jun 13, 2016 at 4:35 AM, Daniel P. Berrange  
> wrote:
> > On Thu, Jun 09, 2016 at 05:31:13PM -0600, Carl Baldwin wrote:
> >> Hi,
> >>
> >> You may or may not be aware of the vlan-aware-vms effort [1] in
> >> Neutron.  If not, there is a spec and a fair number of patches in
> >> progress for this.  Essentially, the goal is to allow a VM to connect
> >> to multiple Neutron networks by tagging traffic on a single port with
> >> VLAN tags.
> >>
> >> This effort will have some effect on vif plugging because the datapath
> >> will include some changes that will effect how vif plugging is done
> >> today.
> >>
> >> The design proposal for trunk ports with OVS adds a new bridge for
> >> each trunk port.  This bridge will demux the traffic and then connect
> >> to br-int with patch ports for each of the networks.  Rawlin Peters
> >> has some ideas for expanding the vif capability to include this
> >> wiring.
> >>
> >> There is also a proposal for connecting to linux bridges by using
> >> kernel vlan interfaces.
> >>
> >> This effort is pretty important to Neutron in the Newton timeframe.  I
> >> wanted to send this out to start rounding up the reviewers and other
> >> participants we need to see how we can start putting together a plan
> >> for nova integration of this feature (via os-vif?).
> >
> > I've not taken a look at the proposal, but on the timing side of things
> > it is really way to late to start this email thread asking for design
> > input from os-vif or nova. We're way past the spec proposal deadline
> > for Nova in the Newton cycle, so nothing is going to happen until the
> > Ocata cycle no matter what Neutron want  in Newton. For os-vif our
> > focus right now is exclusively on getting existing functionality ported
> > over, and integrated into Nova in Newton. So again we're not really looking
> > to spend time on further os-vif design work right now.
> >
> > In the Ocata cycle we'll be looking to integrate os-vif into Neutron to
> > let it directly serialize VIF objects and send them over to Nova, instead
> > of using the ad-hoc port-binding dicts.  From the Nova side, we're not
> > likely to want to support any new functionality that affects port-binding
> > data until after Neutron is converted to os-vif. So Ocata at the earliest,
> > but probably more like P, unless the Neutron conversion to os-vif gets
> > completed unexpectedly quickly.
> 
> In light of this feature being requested by the NFV, container and
> baremetal communities, and that Neutron's os-vif integration work
> hasn't begun, does it make sense to block Nova VIF work? Are we
> comfortable, from a wider OpenStack perspective, to wait until
> possibly the P release? I think it's our collective responsibility as
> developers to find creative ways to meet deadlines, not serializing
> work on features and letting processes block us.

Everyone has their own personal set of features that are their personal
priority items. Nova evaluates all the competing demands and decides on
what the project's priorities are for the given cycle. For Newton Nova's
priority is to convert existing VIF functionality to use os-vif. Anything
else vif related takes a backseat to this project priority. This formal
modelling of VIFs and developing a plugin facility has already been strung
out over at least 3 release cycles now. We're finally in a position to get
it completed, and we're not going to divert attention away from this, to
other new features requests until its done as that'll increase the chances
of it getting strung out for yet another release which is in no ones
interests.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Initial oslo.privsep conversion?

[Daniel P. Berrange]

On Fri, Jun 10, 2016 at 09:51:03AM +1000, Tony Breeds wrote:
> On Fri, Jun 10, 2016 at 08:24:34AM +1000, Michael Still wrote:
> > On Fri, Jun 10, 2016 at 7:18 AM, Tony Breeds 
> > wrote:
> > 
> > > On Wed, Jun 08, 2016 at 08:10:47PM -0500, Matt Riedemann wrote:
> > >
> > > > Agreed, but it's the worked example part that we don't have yet,
> > > > chicken/egg. So we can drop the hammer on all new things until someone
> > > does
> > > > it, which sucks, or hope that someone volunteers to work the first
> > > example.
> > >
> > > I'll work with gus to find a good example in nova and have patches up
> > > before
> > > the mid-cycle.  We can discuss next steps then.
> > >
> > 
> > Sorry to be a pain, but I'd really like that example to be non-trivial if
> > possible. One of the advantages of privsep is that we can push the logic
> > down closer to the privileged code, instead of just doing something "close"
> > and then parsing. I think reinforcing that idea in the sample code is
> > important.
> 
> I think *any* change will show that.  I wanted to pick something achievable in
> the short timeframe.
> 
> The example I'm thinking of is nova/virt/libvirt/utils.py:update_mtime()
> 
>  * It will provide a lot of the boiler plate
>  * Show that we can now now replace an exec with pure python code.
>  * Show how you need to retrieve data from a trusted source on the priviledged
>side
>  * Migrate testing
>  * Remove an entry from compute.filters
> 
> Once that's implace chown() in the same file is probably a quick fix.
> 
> Is it super helpful? does it have a measurable impact on performance, 
> security?
> The answer is probably "no"
> 
> I still think it has value.
> 
> Handling qemu-img is probably best done by creating os-qemu (or similar) and
> designing from the ground up with provsep in mind.  Glance and Cinder would
> benefit from that also.  That howveer is waaay to big for this cycle.

Personally I'd stay away from anything related to libvirt disk file
management / qemu-img / etc. That code is in the middle of being
refactored to use libvirt storage pools, so it is going to change
in structure alot and partially eliminate the need for privileged
command execution. IOW, I don't think it'd make a good example
long term, and the code used for your example may well disappear
real soon.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][os-vif] Expanding vif capability for wiring trunk ports

[Daniel P. Berrange]

On Mon, Jun 13, 2016 at 11:35:17PM +, Peters, Rawlin wrote:
> That said, there are currently a couple of vif-plugging strategies
> we could go with for wiring trunk ports for OVS, each of them
> requiring varying levels of os-vif augmentation:
>
> Strategy 1) When Nova is plugging a trunk port, it creates the OVS
> trunk bridge, attaches the tap to it, and creates one patch port
> pair from the trunk bridge to br-int.
>
> Strategy 2) Neutron passes a bridge name to Nova, and Nova uses this
> bridge name to create the OVS trunk bridge and attach the tap to it
> (no patch port pair plugging into br-int).

[snip]

> If neither of these strategies would be in danger of not making it
> into the Newton release, then I think we should definitely opt for
> Strategy 1 because it leads to a simpler overall solution. If only
> Strategy 2 is feasible enough to make it into os-vif for Newton,
> then we need to know ASAP so that we can start implementing the
> required functionality for the OVS agent to monitor for dynamic trunk
> bridge creation/deletion.

IMHO the answer should always be to go for the right long term architectural
solution, not take short cuts just to meet some arbitrary deadline, because
that will compromise the code over the long term. From what you are saying
it sounds like strategy 1 is the optimal long term solution, so that should
be where effort is focused regardless.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][os-vif] Expanding vif capability for wiring trunk ports

[Daniel P. Berrange]

On Tue, Jun 14, 2016 at 02:10:52AM -0700, Kevin Benton wrote:
> Strategy 1 is being pitched to make it easier to implement with the current
> internals of the Neutron OVS agent (using integration bridge plugging
> events). I'm not sure that's better architecturally long term because the
> OVS agent has to have logic to wire up patch ports for the sub-interfaces
> anyway, so having the logic to make it wire up patch port for the parent
> interface is not out of place.
> 
> Also consider that we will now have to tell os-vif two bridges to use if we
> go with strategy 1. One bridge to create and attach the VM to, and another
> for the other half of the patch port. This means that we are going to have
> to leak more details of what Neutron is doing into the VIF details of the
> neutron port data model and relay that to Nova...

It sounds like strategy 2 also requires you to pass a second bridge
name to nova/os-vif, unless I'm mis-understanding the description
below.


> On Tue, Jun 14, 2016 at 1:29 AM, Daniel P. Berrange 
> wrote:
> 
> > On Mon, Jun 13, 2016 at 11:35:17PM +, Peters, Rawlin wrote:
> > > That said, there are currently a couple of vif-plugging strategies
> > > we could go with for wiring trunk ports for OVS, each of them
> > > requiring varying levels of os-vif augmentation:
> > >
> > > Strategy 1) When Nova is plugging a trunk port, it creates the OVS
> > > trunk bridge, attaches the tap to it, and creates one patch port
> > > pair from the trunk bridge to br-int.
> > >
> > > Strategy 2) Neutron passes a bridge name to Nova, and Nova uses this
> > > bridge name to create the OVS trunk bridge and attach the tap to it
> > > (no patch port pair plugging into br-int).
> >
> > [snip]
> >
> > > If neither of these strategies would be in danger of not making it
> > > into the Newton release, then I think we should definitely opt for
> > > Strategy 1 because it leads to a simpler overall solution. If only
> > > Strategy 2 is feasible enough to make it into os-vif for Newton,
> > > then we need to know ASAP so that we can start implementing the
> > > required functionality for the OVS agent to monitor for dynamic trunk
> > > bridge creation/deletion.
> >
> > IMHO the answer should always be to go for the right long term
> > architectural
> > solution, not take short cuts just to meet some arbitrary deadline, because
> > that will compromise the code over the long term. From what you are saying
> > it sounds like strategy 1 is the optimal long term solution, so that should
> > be where effort is focused regardless.
> >
> > Regards,
> > Daniel
> > --
> > |: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/
> > :|
> > |: http://libvirt.org  -o- http://virt-manager.org
> > :|
> > |: http://autobuild.org   -o- http://search.cpan.org/~danberr/
> > :|
> > |: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc
> > :|
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][os-vif] Expanding vif capability for wiring trunk ports

[Daniel P. Berrange]

On Tue, Jun 14, 2016 at 02:35:57AM -0700, Kevin Benton wrote:
> In strategy 2 we just pass 1 bridge name to Nova. That's the one that is
> ensures is created and plumbs the VM to. Since it's not responsible for
> patch ports it doesn't need to know anything about the other bridge.

Ok, so we're already passing that bridge name - all we need change is
make sure it is actuall created if it doesn't already exist ? If so
that sounds simple enough to add to os-vif - we already have exactly
the same logic for the linux_bridge plugin


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] [nova] os-brick privsep failures and an upgrade strategy?

[Daniel P. Berrange]

On Tue, Jun 14, 2016 at 07:49:54AM -0400, Sean Dague wrote:

[snip]

> The crux of the problem is that os-brick 1.4 and privsep can't be used
> without a config file change during the upgrade. Which violates our
> policy, because it breaks rolling upgrades.

os-vif support is going to face exactly the same problem. We just followed
os-brick's lead by adding a change to devstack to explicitly set the
required config options in nova.conf to change privsep to use rootwrap
instead of plain sudo.

Basically every single user of privsep is likely to face the same
problem.

> So... we have a few options:
> 
> 1) make an exception here with release notes, because it's the only way
> to move forward.

That's quite user hostile I think.

> 2) have some way for os-brick to use either mode for a transition period
> (depending on whether privsep is configured to work)

I'm not sure that's viable - at least for os-vif we started from
a clean slate to assume use of privsep, so we won't be able to have
any optional fallback to non-privsep mode.

> 3) Something else ?

3) Add an API to oslo.privsep that lets us configure the default
   command to launch the helper. Nova would invoke this on startup

  privsep.set_default_helper("sudo nova-rootwrap ")

4) Have oslo.privsep install a sudo rule that grants permission
   to run privsep-helper, without needing rootwrap.

5) Have each user of privsep install a sudo rule to grants
   permission to run privsep-helper with just their specific
   entry point context, without needing rootwrap

Any of 3/4/5 work out of the box, but I'm probably favouring
option 4, then 5, then 3.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] No middle-man - when does/will Nova directly connect iSCSI volumes?

[Daniel P. Berrange]

On Wed, Jun 15, 2016 at 04:59:39PM -0700, Preston L. Bannister wrote:
> QEMU has the ability to directly connect to iSCSI volumes. Running the
> iSCSI connections through the nova-compute host *seems* somewhat
> inefficient.
> 
> There is a spec/blueprint and implementation that landed in Kilo:
> 
> https://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/qemu-built-in-iscsi-initiator.html
> https://blueprints.launchpad.net/nova/+spec/qemu-built-in-iscsi-initiator
> 
> From looking at the OpenStack Nova sources ... I am not entirely clear on
> when this behavior is invoked (just for Ceph?), and how it might change in
> future.
> 
> Looking for a general sense where this is headed. (If anyone knows...)
> 
> If there is some problem with QEMU and directly attached iSCSI volumes,
> that would explain why this is not the default. Or is this simple inertia?
> 
> 
> I have a concrete concern. I work for a company (EMC) that offers backup
> products, and we now have backup for instances in OpenStack. To make this
> efficient, we need to collect changed-block information from instances.
> 
> 1)  We could put an intercept in the Linux kernel of the nova-compute host
> to track writes at the block layer. This has the merit of working for
> containers, and potentially bare-metal instance deployments. But is not
> guaranteed for instances, if the iSCSI volumes are directly attached to
> QEMU.
> 
> 2)  We could use the QEMU support for incremental backup (first bit landed
> in QEMU 2.4). This has the merit of working with any storage, by only for
> virtual machines under QEMU.
> 
> As our customers are (so far) only asking about virtual machine backup. I
> long ago settled on (2) as most promising.
> 
> What I cannot clearly determine is where (1) will fail. Will all iSCSI
> volumes connected to QEMU instances eventually become directly connected?

Our long term goal is that 100% of all network storage will be connected
to directly by QEMU. We already have the ability to partially do this with
iSCSI, but it is lacking support for multipath. As & when that gap is
addressed though, we'll stop using the host OS for any iSCSI stuff.

So if you're requiring access to host iSCSI volumes, it'll work in the
short-medium term, but in the medium-long term we're not going to use
that so plan accordingly.

> Xiao's unanswered query (below) presents another question. Is this a
> site-choice? Could I require my customers to configure their OpenStack
> clouds to always route iSCSI connections through the nova-compute host? (I
> am not a fan of this approach, but I have to ask.)

In the short term that'll work, but long term we're not intending to
support that once QEMU gains multi-path. There's no timeframe on when
that will happen though.



Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] libvirt driver: who should create the libvirt.xml file?

[Daniel P. Berrange]

On Mon, Jun 20, 2016 at 05:47:57PM +0200, Markus Zoeller wrote:
> White working on the change series to implement the virtlogd feature I
> got feedback [1] to move code which creates parts of the libvirt.xml
> file from the "driver" module into the "guest" module. I'm a bit
> hesitant to do so as the responsibility of creating a valid libvirt.xml
> file is then spread across 3 modules:
> * driver.py
> * guest.py
> * designer.py
> I'm only looking for a guideline here (The "driver" module is humongous
> and I think it would be a good thing to have the "libvirt.xml" creation
> code outside of it). Thoughts?

The designer.py file was created as a place which would ultimately hold
all the XML generator logic.

Ultimately the "_get_guest_xml" (and everything it calls) from driver.py
would move into the designer.py class. Before we could do that though, we
needed to create the host.py + guest.py classes to isolate the libvirt
API logic.

Now that the guest.py conversion/move is mostly done, we should be able
to start moving the XML generation out of driver.py  and into designer.py

I would definitely *not* put XML generation code into guest.py

In terms of your immediate patch, I'd suggest just following current
practice and putting your new code in driver.py.  We'll move everything
over to designer.py at the same time, later on.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] No middle-man - when does/will Nova directly connect iSCSI volumes?

[Daniel P. Berrange]

On Thu, Jun 23, 2016 at 09:09:44AM -0700, Walter A. Boring IV wrote:
> 
> volumes connected to QEMU instances eventually become directly connected?
> 
> > Our long term goal is that 100% of all network storage will be connected
> > to directly by QEMU. We already have the ability to partially do this with
> > iSCSI, but it is lacking support for multipath. As & when that gap is
> > addressed though, we'll stop using the host OS for any iSCSI stuff.
> > 
> > So if you're requiring access to host iSCSI volumes, it'll work in the
> > short-medium term, but in the medium-long term we're not going to use
> > that so plan accordingly.
> 
> What is the benefit of this largely monolithic approach?  It seems that
> moving everything into QEMU is diametrically opposed to the unix model
> itself and
> is just a re-implementation of what already exists in the linux world
> outside of QEMU.

There are many benefits to having it inside QEMU. First it gives us
improved isolation between VMs, because we can control the network
I/O directly against the VM using cgroup resource controls. It gives
us improved security, particularly in combination with LUKS encryption
since the unencrypted block device is not directly visible / accessible
to any other process. It gives us improved reliability / managability
since we avoid having to spawn the iscsi client tools which have poor
error reporting and have been frequent sources of instability in our
infrastructure (eg see how we have to blindly re-run the same command
many times over because it randomly times out). It will give us improved
I/O performance because of a shorter I/O path to get requests from QEMU
out to the network.

NB, this is not just about iSCSI, the same is all true for RBD where
we've also stopped using in-kernel RBD client and do it all in QEMU.

> Does QEMU support hardware initiators? iSER?

No, this is only for case where you're doing pure software based
iSCSI client connections. If we're relying on local hardware that's
a different story.

> 
> We regularly fix issues with iSCSI attaches in the release cycles of
> OpenStack,
> because it's all done in python using existing linux packages.  How often

This is a great example of the benefit that in-QEMU client gives us. The
Linux iSCSI client tools have proved very unreliable in use by OpenStack.
This is a reflection of the very architectural approach. We have individual
resources needed by distinct VMs, but we're having to manage them as a host
wide resource and that's creating us unneccessary complexity and having a
poor effect on our reliability overall.

> are QEMU
> releases done and upgraded on customer deployments vs. python packages
> (os-brick)?

We're removing the entire layer of instability by removing the need to
deal with any command line tools, and thus greatly simplifying our
setup on compute nodes. No matter what we might do in os-brick it'll
never give us a simple or reliable system - we're just papering over
the flaws by doing stuff like blindly re-trying iscsi commands upon
failure.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][pci-passthrough] definitely specify VFIO driver as the host PCI driver for passthrough

[Daniel P. Berrange]

On Fri, Jun 24, 2016 at 12:27:57PM +0800, Chen Fan wrote:
> hi all,
>  in openstack, we can use the pci passthrough feature now, refer to
>  https://wiki.openstack.org/wiki/Pci_passthrough
>  but we can't definitely specify the host pci driver is LEGACY_KVM or
> newer VFIO,
>  new VFIO driver is more safer, and higher-performance user-space driver
>  than legacy kvm driver (pci-stub), the benefit relative to kvm
> assignment driver
>  could refer to http://lwn.net/Articles/474088/.
>  In additional, VFIO driver provides the GPU passthrough as primary card
> support.
>  I think it is more useful for further GPU passthrough support in
> openstack.
> 
>  Openstack relies on the libvirt nodedev device configuration to do pci
> passthrough,
>  with managed mode, the configured device is automatically detached and
> re-attached
>  with KVM or VFIO driver that depended on the host driver modules
> configuration,
>  so now we can't specify the driver in openstack to VFIO mode, I think
> we should need
>  to add this feature support in openstack to get pci passhthrough more
> scalability.
> 
>  a simply idea is to add a option in nova.conf HOST_PCI_MODEL = VFIO
> /KVM to specify
>  the pci passthrough device driver is using VFIO driver.
>  any comments are welcome. :)

I don't see any reason to add a configuration option. If the host is
capable of doing VFIO, libvirt will always aim to use VFIO in preference
to the legacy system.


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev