[Openstack] TC candidacy
I'd like to hereby nominate myself as candidate for a seat on the Technical Committee. I'm a Senior Software Engineer at Cisco Systems. In the past, I've held similar positions at Nebula, Rackspace and Canonical. I've been part of this project since before it was called OpenStack, mostly focusing on Nova, but also spending a lot of time building up a lot of the automation (including the CI infrastructure) to help us hit the ground running in the very beginning. Monty's team is doing a fantastic job looking after the infrastructure stuff now, so I've almost entirely backed off from that. I'm a core developer of Ubuntu and hold a seat on Ubuntu's technical board. In the course of my work on Ubuntu, I've contributed to countless open source projects, in recent years mostly focusing on virtualisation related things. I think our most important qualities in OpenStack are reliability and scalability and this will very likely shine through in my work on the technical committee. Even though I've spent most of my OpenStack time working on Nova, I have a keen sense of the bigger picture, both in terms of considering the other OpenStack projects, but also considering our various upstreams and downstreams. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OSSA 2012-014] Revoking a role does not affect existing tokens (CVE-2012-4413)
So if I can grant people access to a particular tenant, I can invalidate everyone's tokens at will now? Best regards, Soren. Sent from my phone. Please pardon my brevity. On Sep 12, 2012 6:40 PM, "Thierry Carrez" wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > OpenStack Security Advisory: 2012-014 > CVE: CVE-2012-4413 > Date: September 12, 2012 > Title: Revoking a role does not affect existing tokens > Impact: High > Reporter: Dolph Mathews (Rackspace) > Products: Keystone > Affects: Essex, Folsom > > Description: > Dolph Mathews reported a vulnerability in Keystone. Granting and > revoking roles from a user is not reflected upon token validation for > pre-existing tokens. Pre-existing tokens continue to be valid for the > original set of roles for the remainder of the token's lifespan, or > until explicitly invalidated. This fix invalidates all tokens held by > a user upon role grant/revoke to circumvent the issue. > > Folsom fix: > > http://github.com/openstack/keystone/commit/efb6b3fca0ba0ad768b3e803a324043095d326e2 > > Essex fix: > > http://github.com/openstack/keystone/commit/58ac6691a21675be9e2ffb0f84a05fc3cd4d2e2e > > References: > https://bugs.launchpad.net/keystone/+bug/1041396 > http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=2012-4413 > > Notes: > This fix will be included in the future Keystone 2012.1.3 stable > update and the upcoming Folsom-RC1 development milestone. > > - -- > Thierry Carrez (ttx) > OpenStack Vulnerability Management Team > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ > > iQIcBAEBCAAGBQJQULoUAAoJEFB6+JAlsQQjGacQAJUvJb+oIjh73KAYYuDpl/YP > PqJa4nmjVin7CyQ8AbxHK63xrAQ7isPFpCCqtEmjZ5kvFCrJRHiQggHNqISRhnvo > +HyS6RSn4Vrp001PSZSmQI5MpgkeWhbOy+fk4/ZY7hFgUyS2YqC8YiK7DTMdKRBi > toWOHRVWrmA4fUEDDcDdm9XzRseTC0cZAbj9bYAF+vXPdpxeGpq5l9Kb6yDezXGD > 62dFvHghVTWdUIN+gK4V4d77PoyeO9NRd4Ud0GjDpV/asQL31dW6B4aRPYVDPhL3 > 7xcnhRsnZ3Y5J31n+7E/gMF+J+6kOaY/DNFZQ8chNW18kplYnmJnm7s3BJNjD512 > UF/S5A5sH1Rk/vwe2nAHSqvQ1Dq3K0sRvW3YCijG2Rdj3mhBOr6OlvT5uJmnkeJT > GQQ8SR3y+ZLS/2EEW+cVjDMxV4Gnf9Zzrw/tSjVp6QLmJAkG8qrFmgdisQ/Jao4M > ygE8ZVu8lJq7N8b+k8XkB+bhz9E9V6hYOUuGoifEHRIPki/Ed7++BcdVTQdQYpAL > kDTaoVZt1+plwAu4ZBLxUg1vhVz19qgDc7UeoY1sPc1JcRWp/ONnp6K4z+Y+7Rsx > 3E4FLH0/qgFxKDHdGX91Plehk9dIEjHcGtKaXI8vOvGT17srYQaF6Y7rc+9TwaqI > bggBCxcI2PLQgjuWyF4M > =+6UN > -END PGP SIGNATURE- > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack Foundation] Individual Nominations for Foundation Board of Directors
Nope, that's perfectly clear, I just failed to spot it. Thanks. Best regards, Soren. Sent from my phone. Please pardon my brevity. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack Foundation] Individual Nominations for Foundation Board of Directors
Sorry if it's already been made clear somewhere, but I'm unsure for what term these individuals are being elected? The bylaws say that the election happens each year in the first two weeks of January, so is this an election to fill the seats until the first ordinary election in January? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova] [cinder] Nova-volume vs. Cinder in Folsom
Apologies if this has already been proposed, but how about an option 3 (perhaps more accurately option 2.5): We already have a process for maintaining code in one project and occasionally copying it to another project. Namely, code is maintained in openstack-common and then -- at appropriate times -- gets copied to Nova or Glance or whatever. Correct? Seeing as Cinder is supposedly a straight copy of Nova volume, it seems feasible to occasionally copy it all back into Nova. This way, it's not a matter of "fixing bugs (and adding features and whatever) twice", but rather fixing bugs (and adding features and whatever) once and the rest is straight-forward (possibly even easily scriptable) patch management. Obviously, this wouldn't happen indefinitely, but simply serve to bridge the gap between those who want to split it out (with which I can certainly sympathise) and those who want to keep it Nova for Folsom (which I can also sort of relate to). -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Greatest deployment?
2012/5/30 Matt Joyce : > On Wed, May 30, 2012 at 12:39 PM, Soren Hansen wrote: >> 2012/5/30 Matt Joyce : >>> Secondly, while LXC does provide a lot of native access, it still >>> does paging management internally just as kvm does. So direct >>> memory management ( some HPC users like this ) becomes just as >>> problematic as it is in kvm. Lots of overhead. >> I'm not convinced this is accurate. Can you provide some kind of >> reference for this? > Okay so KVM uses a nastier abstraction layer in the form of shadow > paging, while LXC simply relies on cgroups for memory isolation. > Obviously two very different beasts. But there is the overhead of > cgroup accounting and resource management inside LXC. Ah, yes. Cgroups. I'm obviously behind the times. In my head, an LXC container is just a namespaced (set of) process(es). Very good point. Thanks. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Greatest deployment?
2012/5/30 Matt Joyce : > Secondly, while LXC does provide a lot of native access, it still does > paging management internally just as kvm does. So direct memory management > ( some HPC users like this ) becomes just as problematic as it is in kvm. > Lots of overhead. I'm not convinced this is accurate. Can you provide some kind of reference for this? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-compute] vm migration problem
2012/5/21 Lorin Hochstein : > Has anybody ever written a script that grabs the host public key from > the instance's console and updates the .ssh/config/known_hosts file > accordingly, instead of throwing away host key checking? That would > be a handy little thing if it was out there. Ubuntu's cloud-utils package has a cloud-run-instances utility that does this. It's not exactly in the do-one-thing-and-do-it-well sort of category, but perhaps it's just what you need. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] swift news and plans
2012/5/15 Andy Edmonds : > If I'm not mistaken: > http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-05-08-20.00.log.html That meeting happened 4 days *after* the e-mail I responded to was sent to the mailing list. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] swift news and plans
2012/5/4 John Dickinson : > TL;DR: removing code from swift, associated projects doc, swift 1.5.0 This is interesting stuff. Where was this discussed? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Core Cleanup
2012/5/8 Monsyne Dragon : > Just wondering, who is maintaining the review day schedule? > (http://wiki.openstack.org/Nova/ReviewDays) I can easily make time for more > reviews if I tell folks it's my review day, but I have not seen my name on > the schedule there for several weeks. I do. I have a script that takes care of it, but for various reasons (the fact that some people don't have any public e-mail adresses on Launchpad being high on the list) I have to maintain it manually. It would appear that you've somehow managed to escape the list. :) I'll rectify that the next time I generate the table. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
23. apr. 2012 17.15 skrev Justin Santa Barbara : >> What's the advantage of replacing the native EC2 compatibility layer with >> AWSOME from a user / operator point of view? > Although I wasn't able to attend the design summit session, right now we > have two "native" APIs, which means we have two paths into the system. Yes. This is good. It keeps the layers of separation "honest". Multiple consumers of internal API's helps in making it more obvious at which layer functionality belongs. Just like having multiple hypervisor drivers (at least supposedly) makes it more obvious what sort of stuff belongs in the drivers and what sort of stuff belongs in the compute manager. > That is poor software engineering, because we must code and debug > everything twice. I beg to differ. If we need to fix things twice, it's because we've violated the layers of separation somewhere. For the record, I'm fully prepared to believe that we've done so. I also fully believe that the fact that we haven't done so *more* is because we have two API's to consider. Eric Day went through all the calls from the frontends into the various backend managers a long time ago, ensuring this separation was clear. I'm convinced that the result was a massive improvement. > Some developers will naturally favor one API over the other, and so > disparities happen. Today, both APIs are effectively using an > undocumented private API, which is problematic. Yes. This is a problem. However, as I understand it, there was a session at the summit about versioning the internal API's. I'm not sure how we can usefully version the API's without also documenting them, so that problem should be adressed in the relatively near future. > We also can't really extend the EC2 API, so it is holding us back as > we extend OpenStack's capabilities past those of the legacy clouds. If EC2 API is limiting what we can do, that's not going to change just because you move the EC2 API implementation into a proxy in front of the OpenStack API. The only difference is that it's suddenly the AWSOME development team's problem. > With one native API, we can focus all our energies on making sure that API > works. Then, knowing that the native API works, we can build other APIs on > top through simple translation layers, and they will work also. Other APIs > can be built on top in the same way (e.g. OCCI) Sorry, I'm having trouble here.. Are you suggesting that having two sibling frontend API's talking to a shared backend API is poor software engineering, but layering similar purposed API's on top of each other like this is good software engineering? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] I18n issue for OpenStack
Don't get me wrong.. I'd be happy to have the various openstack clients offer localised error messages. I'd also encourage a centralised effort to collect these translationns (so that all the various language bindings will use the same localised error messages). On the server, though, I believe we should stick to English and perhaps have every error message include a link (e.g. http://docs.openstack.doc/exception/NoNetworksDefinedException) to a localised docs site. I think losing the ability to search the web for error messages would be a major loss. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] I18n issue for OpenStack
Den 12. apr. 2012 13.27 skrev Hua ZZ Zhang : > These messages always need to be localized, > returned and displayed on user interface, not just be logged in backend > system. It is very common practice for a global software project. OpenStack isn't an interactive application. It's a piece of server software. I'm struggling to think of any server software that writes anything in its log files in anything other than English. I'm Danish and generally configure my servers to speak Danish, so I should have seen it at least occasionally. Do you have any examples of this alleged "very common practice"? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] EC2 metadata
29. mar. 2012 00.55 skrev Joshua Harlow : > I’ve seen that before. But was wondering if there was any secret other wiki. There is no secret wiki that I'm aware of. If you believe the referenced wiki page is out of date, please help update it. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] speed up unittest in Nova
2012/3/9 Hengqing Hu : > Last few weeks I spend some time to speed up unittest in Nova. This is great stuff! Thanks! -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Get nova diagnostics to work with libvirt
2012/3/12 Vishvananda Ishaya : > Diagnostics are a bunch of key/value pairs, but unfortunately there doesn't > seem to be an example of what they are. If someone running xen could post > the result of get_diagnostics so that it could be added to a docstring and > implemented for libvirt, that would be awesome. Once someone works out what this is actually supposed to look like, please add a unit test to test_virt_drivers to document it. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Gerrit minimum review time frame
2012/3/13 John Garbutt : > One extra concern. Since I work in the UK, most things happen while I > am sleeping. > > Not sure I know of a good solution to that problem. A 12 hour window > seems stupidly long, No. No, it doesn't. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Gerrit minimum review time frame
2012/3/13 Jesse Andrews : > Maybe it is just me, but most reviews seem to take hours to days to > complete. I'm been sitting here waiting for a one line change to get > a second "+2 / approved" so I can redeploy our test cluster for the > last 2 hours. Can we keep this discussion separate, please? I acknowledge it can be difficult getting stuff approved (or even reviewed), but that's a separate (yet just at real) problem. > Do we need a "time-gate" or can you use a feed-reader and > https://github.com/openstack/nova/commits/master.atom to read patches > were approved while one is away/sleeping/time off? We have the review process so that people get a chance to disagree before changes land. Otherwise, we could just let nova-core commit directly to trunk and deal with problems after the fact. > Delaying by 2 hours on a weekday at noon vs 2am saturday night? > Would we then start debating how best to gate at different times based > on when people are available? It's always going to be noon or 2 am somewhere. I don't think we need different delays for different times. I could easily be pursuaded to the delay being way longer than 2 hours, though :) -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Gerrit minimum review time frame
2012/3/12 Josh Kearney : > Is this is really a problem that needs solving? I'd like to believe > that no member of Nova Core would approve something that they aren't > familiar with. That's not the point. The review process isn't just about finding enough people who agree with your change, it's just as much (some might argue, it's even more) about giving people a chance to *disagree*. > IMHO, we should be making better attempts at not letting branches sit > around for days/weeks at a time. I agree. That's a completely separate issue, though. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] cobbler
2012/3/12 khabou imen : > I want to use cobbler between 2 server from ovh both of them has > ubuntu server 11.10 as os in the first I installed orchestra and > > when I use cobbler to boot in PXE mode to the next machine i failed > to use juju (the commande juju status didn't work) > > > My question is :Is'it possible to boot in PXE mode on a machine which > has ubuntu-sever 11.10 as OS or I must have a VM to be able to use > cobbler Whether or not you can PXE boot is not dependent on the OS. It depends on the hardware (and firmware). -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Can openstack support spice ?
2012/3/11 wangsuyi640 : > Dear all: > > The release of openstack on my server is D3. I tred to run ' > qemu-kvm -hda /root/free.img -m 512 -vga qxl -spice > port=5930,disable-ticketing ' > > without openstack , it worked well. > > However,I have tried to modify the > '/opt/stack/nova/nova/virt/libvirt.xml.template' in order to let the spice > work with the openstack. Then it failed. > > Is there anyone tried this? Could you give me some help? Thanks. It's a bit hard to work out what is failing when you don't tell us what you actually changed. What does your changed libvirt xml template look like? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Libvirt Snapshots
2012/3/9 Vishvananda Ishaya : > OPTION B --> libvirt 9.5 snapshots > > This method uses the newer snapshot xml in libvirt 9.5 to snapshot only the > root disk. > > Pros: > plays nicely with libvirt, so the vm is only paused for the minimum amount > of time > Cons: > requires libvirt 9.5, which doesn't exist in oneiric Oneiric is old hat. I'm cool with 0.9.5. If someone wants to spend time building an inferior (i.e. that requires more downtime during snapshotting) implementation that works with older versions of libvirt, that's fine, but as a project, my view is: There's excellent free software available that enables us to build cooler things faster. We should use it. Besides, libvirt 0.9.5 has been out for over 6 months. It's not *that* new. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Running for Nova PTL
2012/2/28 Vishvananda Ishaya : > There is always progress to be made, but while we are toiling away working > on testing, technical debt, and code smell, we have to keep our users in > mind. This part of your e-mail really sticks out, along with this one: > I disagree with this point. There was a huge amount of effort from a lot of > parties on getting things cleaned up during essex. Most of the "features" > that were worked on were getting things aligned and consistent behind our > api. We have to make sure that Nova is usable as well as stable. It is > tempting to think of things from the developer/code perspective, but we have > to put on our user/operator hats as well. I'm trying hard to work out what you're implying here.. Are you suggesting that my suggested policy of not adding features, but focusing almost exclusively on stability isn't for the best of our users? Who do you think (I think) I'm favouring when I propose something like that? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Running for Nova PTL
2012/2/24 Eric Windisch : > On Friday, February 24, 2012 at 12:01 PM, ed_con...@dell.com wrote: > >> That can work and may be the only choice if there is an extended feature >> freeze. Although, that may end up creating a service provider-specific >> fork...which may not be a bad thing. > It can also be a very, very bad thing. Segmentation of the community > and an exponentially increased complexity for those of us playing both > sides of the private/public fence. I really can't see any advantage > of forking, even temporarily. "Forking" has a number of connotations, not all of which necessarily strictly apply here. We have some fundamental problems we need to address. To do so effectively while still letting others move forward towards their goals, the proposal (in its current form) is to let people work on the things they need to meet their goals in a separate branch (probably in a separate repository), yet still under the Nova umbrella. Once we feel we've addressed the aforementioned problems, we can start merging the features in from these other branches. As I said, I'm expecting these "experimental" (or whatever designation is appropriate) branches will frequently be rebased on top of trunk so as to ease a later merge of the branches. The fact that these "forks" are still in many ways part of the project (albeit do not follow the rigour of the OpenStack release cycle), as well as the fact that they are expected to rebase often, make them fail to qualify as "forks", at least in the traditional sense. > I'm opposed to a broad Folsom feature freeze as it would too greatly > limit progression. However, I also agree that there needs to be a > better core focus, rather than on sprawl. I don't believe what Nova needs is features, be they "core" or in the various drivers. We need stability and dependability. > I'm not opposed to selective feature inclusion. On the same token, I > believe we should either approach the Linux kernel model of "include > the kitchen sink" or not, and by not, Nova would be the core framework > upon which various drivers would be provided and could be plumbed. Would you mind proposing this as a session for the summit? It sounds like a good chat to have face to face. One of the things I'd like for us to address in Folsom is actually to make this easier. Many parts of the nova code base can't easily be split out, but drivers should be fairly easy to maintain separately. We should make that simpler. This, like other things I've suggested, is mostly a job of clarifying contracts between various components. > If today, for instance, it was announced that Folsom won't include new > features, then it would be impossible for Coraid, Pillar, or some > other storage solution provider to offer a driver in Folsom and would > have wait until G. Yet, Nexenta just got their driver into Essex. > Nexenta's 6 month lead just turned into a 12 month lead! Sure, their > competitor could ship separately, but there *is* a difference between > inclusion, and now, if only politically and from the perspective of > marketing. That is a very good point, and like all good policies, exceptions can be made. A change that simply adds a new driver (and doesn't really touch any core parts of Nova at all) would be a very good candidate for an exception, particularly in the volume subsystem, which, last I looked at it, seemed quite well behaved. > If new drivers and new code won't be accepted easily, then these third > party drivers should be maintained as external plugins. While nice in > theory, I don't agree with it at this time. These are contributions to > OpenStack and are core, essential to the success of the community. > Breaking out drivers, while easier, would fracture the community in > potentially devastating ways. That's a valid point of view. However, when we need to weigh it against the burden of maintaining potentially unmaintainable plugins, it's not always an easy decision. We'd certainly need to clarify the social contracts better than we have in the past. So, in case we have to accept into core a driver for some piece of infrastructure we cannot easily get access to (and as a team don't really have a vested interest in), someone needs to be "all over it". We need to make sure those expectations are clearly communicated. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Running for Nova PTL
2012/2/24 : > That can work and may be the only choice if there is an extended feature > freeze. Although, that may end up creating a service provider-specific > fork...which may not be a bad thing. That's certainly not my intent nor desire. At least not permanently. I'd love to have a chat about the details of how we can make this work both short and long term. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Running for Nova PTL
2012/2/24 : > I like most of what you say, but the ""no new features at all" policy > for trunk for the Folsom" causes concern. I absolutely agree that Nova > needs to be more stable, predictable and have more work done around > operations. This is especially true for service provider usage. I > think if Nova was closer to feature complete, that the ""no new > features at all" policy for trunk for the Folsom" would be fine. It is > probably doable for Folsom if the focus is on private cloud > functionality. But for service providers, Nova is not feature complete > and there is still much needed from the operations standpoint. I understand it's a trade-off. People may feel there are features missing before they can really use Nova, but noone can use it if it's unstable or doesn't work. I believe everyone will be better off if we spend some time (like, say, 6 months time) and focus exclusively on addressing these things. Thinking we can both add lots of new features as well as rework things for more stability at the same time is how we've ended up here. If you know of a way to achieve both, I'd love to hear about it. As much as I wish everyone would drop what they are doing and start helping in this effort, I understand some groups have certain features they simply cannot defer. I'd like to encourage this to happen collaboratively in public repos, and with frequent rebases against trunk to ease inclusion into trunk later on. In a way, they'd be patch sets more than anything else. How does that sound to you? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Basic networking/configuration woes
2012/2/24 Justin Santa Barbara : > I have contributed a patch (which has merged) which should allow you > to stop editing the SQL: https://review.openstack.org/#change,3816 > With that, you should be able to pass the full range, with an > additional argument specifying the subset that nova controls: > e.g.-fixed_cidr=10.200.0.0/16 > > When I boot my VM, I think it gets a real address from my DHCP server > (because the VM can reach the DHCP server), but not the address nova > assigned it! I believe the nova iptables rules mean that the machine > can't then do TCP/IP, but even if I am wrong/could overcome that, I > don't think cloud-init could then configure the correct address. The instance firewall should be configured to only allow DHCP responses from the IP it believes to be the correct DHCP server. Perhaps it has the wrong idea? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Running for Nova PTL
x and then in > discussions at the next summit) Sure. I don't expect to be able to wave a magic wand, flip the "be stable" bit and everything will suddenly be awesome and solid. > The work done by mtaylor & jblair on gating merges has lead to a much > saner trunk. During diablo our team would routinely spend a few hours > a day fixing trunk. During Essex the timeframe having a broken trunk > was the exception! We've certainly made great strides on that front. However, we need better and more tests to ensure that different combinations of configuration options behave as expected. It's unfair to defer that to deployers who care about said combination of configuration options, and require that they offer ressources to test things there. There are plenty of things we can do to verify that e.g. a volume driver upholds the contract, and that, say, the virt drivers are happy as long as said contract is upheld. I know it's an iterative process, but that doesn't mean the problems are any less real. > I look forward to further discussions about improving openstack > regardless of who is PTL. As do I. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Security Group Rule Refresh
2012/2/23 Day, Phil : >> 1 deal with the situation where a refresh call to one of the compute >> nodes got lost. If that happened, at least it would all get sorted >> out on the next refresh. > Can see the advantage of this, but on an active system this can be > quite an overhead compared to a periodic refresh. Well, a periodic refresh will likely happen more often than the refreshes triggered by changes, don't you think? And periodic refreshes will inevitably have to refresh everything (otherwise they seem pointless). >> 2 the routine that turned the rules from the database into iptables >> rules was complex enough as it was. Making it remove only rules for a >> single security group or a single instance or whatever would make it >> even worse. > I wonder if we're talking about the same driver - the code we're > looking at is in the IptablesFirewallDriver in libvirt/firewall.py > (which I think is moved up to virt/firewall.py in Essex). That seems > to create a chain per Instance and do the update on a per instance > basis, so I'm not quite sure I understand your point ? Sorry, I was basing this all on memory. The point is simply that if the routine that did all of this would have to reliably leave everything else alone, and only touch the rules pertaining to a particular instance, the logic would be even more complicated than it already is. >> 3 The difference in terms of efficiency is miniscule. iptables >> replaces full tables at a time anyway, and while the relative >> amount of data needed to be fetched from the database might be much >> larger than with a more selective refresh, the absolute amount of >> data is still pretty small. > It may be that we're hitting a particular case - but we have a test > system with 10's of VMs per host, on not many hosts, and some groups > with 70+ VMs and a rule set that references the security group itself. > So every VM in that group that gets refreshed (and there are many on > each host) has to rebuild rules for each VM in the group. That's a bug. It's supposed to only refresh once, regardless of how many affected VM's there are. > The impact of this overhead on every VM create and delete in > un-related groups is killing the system - eps as the update code > doesn't yield so other tasks on the compute node (such as the create > itself are blocked). Have you been able to profile this at all? Is it the DB query that takes a long time or is it something else? Anyways, I don't fully understand why any part of the process would make anything hang. Both the communication with the DB as well as calling out to iptables-restore should yield control over to the eventlet main loop and let other things run. I wonder why this isn't happening. >> Point 2 should be more palatable now that the simpler implementation >> has proven itself. > Could you clarify which simpler implementation your referring to It's probably a poor choice of words :) The "simpler" implementation is the current one. The more complicated one would be one that reliably would only touch the rules pertaining to the instances or security groups that are actually being changed. > - I've seen the NWFilterFirewall class and its associated comment > block, but it wasn't clear to me under what circumstances it would be > worth switching to this ? None, at the moment, due to this bug: https://bugzilla.redhat.com/show_bug.cgi?id=642171 -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Running for Nova PTL
2012/2/23 Duncan McGreggor : > Soren, if elected, by what processes/policies etc. would you > accomplish these goals? Well, there are limits to what a PTL really can do :) However, in practical terms there are a number of things I'd like us to do: * I'd like us to look at the various components of Nova and thoroughly document (in prose as well as as tests) their API and expected behaviour. It's very tempting to change (in major or minor ways) these API's on a whim since we control both ends of the channel (often even in the same patch), but this a distributed system. Upgrades across an entire Nova installation are not instantaneous, and shouldn't have to be. We need to be more aware of the interfaces between components and the fact that they may not be in perfect sync. * In a similar vein, while we do a good job ensuring db schema upgrades work well, the code doesn't support anything other than the newest schema it knows about. Or rather, if it does, it's by accident. This makes it exceedingly difficult to upgrade a Nova installation peacemeal. * I'd like to revamp the virtualisation subsystem to move much more behavioural logic into a superclass and have the actual drivers be only the glue code to make the individual hypervisors work. * As I wrote in my response to Robert earlier in this thread, I'd like to see more branches pop up specific to particular subsystems. I'd like to make it easier to get features landed somewhere and let them mature there before they hit trunk. * I'd like to have much more frequent releases, and I mean *actual* releases, not just milestones. Each with merge windows, QA phases, release artifacts, etc. * Lots of other things I'll try to elaborate on over the next few days. > Are there blueprints that already exist which > you would rally folks around? Or would you introduce a new effort to > more thoroughly componentize OpenStack? > > More specifically, how do you envision: > > 1) clarifying what needs to be done I don't expect to do this all on my own. I'd like to set some overall topics for the release cycle and try to seed the conversations about these efforts (as I'm trying to do right now), but I'd really, really like for everyone else to help identify all of this stuff and find issues you care about. > 2) building consensus around this, and Excellent question. I can't force anyone to suddenly think QA and unit tests are the most important things in the world. :) I think there's a strong correlation between my chances of getting elected and the how much of a pre-existing consensus there is around the issues (and issues like them). If I get elected, it's because people already think these things are important, so it shoulnd't be too hard. Or so I hope. > 3) accomplishing these goals? (it's a lot of work!) I hope the rest of my e-mail sheds a bit of light on this. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Running for Nova PTL
I've put my name on the ballot for Nova PTL, and I'd like to explain what I expect to do (my platform, if you will). Nova is facing many separate, but related problems. * Nova is too big. Very few (if any) core developers are comfortable reviewing every part of the code base. In itself, this isn't necessarily a problem, but I think it would be valuable to try to somehow acknowledge that the average focus is much narrower than "all of nova". * Lots of things in Nova that should be orthogonal are not. This problem is especially prevalent in the virtualisation layer. The layout and number of disks you get attached to instances shouldn't depend on the hypervisor you've chosen, but it does. There is lots and lots and lots of logic embedded in both the libvirt and XenServer drivers that isn't related to the hypervisor, but is a result of the origin of these drivers. * The overall quality is decreasing There's an almost unilateral focus on features across the board. The topic of almost every session at the summit is some new feature. There is very little focus on stability, predictability and operation. Personally, I think that shows very clearly in the final product. I'd like to try to shift our focus and turn the proverbial ship around. I'd like to remove any incentive to rush things into Nova trunk. 1. A much shorter release cycle (as Thierry also suggests[1]) would be very beneficial. Noone wants to have to wait an extra 6 months getting some new feature in just because it missed the feature freeze. However, just a single month of delay... That should be manageable in most cases. 2. I'd like to make it more straight forward to have things mature somewhere separete from Nova trunk, but still make it easy to collaborate on them or get people to test them. 3. I'd like to encourage a stronger focus on QA and testing. Specifically, I'd love to have more people focused on making it easier to test things in Nova. Tempest is a great effort, but the unit test suite is our first line of defence. It should be fast and comprehensive. Right now, it's neither. 4. I'd like a stronger focus on extensibility and plugability. 5. I'd like us to rethink our configuration management strategy. So far, we've punted on it and deferred to deployers to choose between Puppet, Chef or whatever else to handle this. However, many things will crash and burn if the configuration of various components is out of sync with each other or with the database. This is particularly clear in the networking area. [1]: http://fnords.wordpress.com/2012/02/21/open-dev-releases-quality/ -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Security Group Rule Refresh
2012/2/22 McNally, Dave (HP Cloud Services) : > Currently I’m trying to track how a refresh of the security groups is > handled (upon creation or deletion of a vm). Following through the > code I get to ‘do_refresh_security_group_rules’ in > libvirt/firewall.py. Up to this point the security group in question > has been carried through however it seems to be discarded here and > rather than filtering the instances to refresh the rules for based on > this group it looks to me like all instances on the current host are > iterated through and then there is an attempt to update the rules for > all these instances. > > Is this full refresh necessary/intentional? If so can anyone tell me > why it’s required? I forget the exact history here (i.e. why some of the method calls include it and why some don't), but there are three reasons I decided to do a full refresh: 1 deal with the situation where a refresh call to one of the compute nodes got lost. If that happened, at least it would all get sorted out on the next refresh. 2 the routine that turned the rules from the database into iptables rules was complex enough as it was. Making it remove only rules for a single security group or a single instance or whatever would make it even worse. 3 The difference in terms of efficiency is miniscule. iptables replaces full tables at a time anyway, and while the relative amount of data needed to be fetched from the database might be much larger than with a more selective refresh, the absolute amount of data is still pretty small. Point 1 could be addressed now by a periodical refresh of the rules, if one was so inclined. Point 2 should be more palatable now that the simpler implementation has proven itself. Point 3 might be less true now. In the beginning, there were separate chains for each security group, now it's just one big list, IIRC. That may change things. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Wish: Please rename all OpenStack packages to openstack-*
2012/2/22 Zhongyue Luo : > Before we discuss whether the packages should be renamed or not, isn't it > unorthodox to have a hyphen in a package name? Out of the 38480 packages available on amd64 in Ubuntu Precise, 28910 have a hyphen in their package name. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Keystone: Redux (Dubstep Remix)
2012/2/14 Jesse Andrews : > The major lessons of keystone: Now that we're verbalising lessons learnt from Keystone, I'd like to add another thing from back in the Diablo days: We should only ever depend on code that already exists or is under our own release management. When Keystone was very young, we deprecated Nova's built-in auth system, but seeing as Keystone wasn't ready, nor was being tracked by our release manager, we ended up releasing Nova with a deprecated auth system and a preferred auth system that wasn't released yet. I'd like to avoid that happening again. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] confused about libvirt nwfilter and iptables rules
The original implementation of this filtering used only nwfilter. Due to shortcomings in nwfilter in libvirt and netfilter in the Linux kernel, this turned out not to work very well at all, so an alternate implementation using raw iptables was added. This is now the default. However, nwfilter works excellently at protecting against MAC spoofing, ARP spoofing and IP spoofing, so we still use it for that. Does that help? 2012/2/7 heut2008 : > hi,all: > I am confued about how security rules works ,i read the > /nova/virt/libvirt/firewall.py and /nova/network/linux_net.py , > my understanding is when create or change a security rule ,the process is > as below. > reuqest to nova osapi->update db for the rule->call method > trigger_security_group_rules_refresh()->rpc.cast to all reletave compute > node. > ->call refresh_security_group_rules(),it seems > that refresh_security_group_rules get the rule from the db and use libvirt > to define the rules. > but how iptables are invoked to create rules "like nova-compute-inst-22". > > anther question is libvirt defines nova-base-filter which allow any > packets out and drop all packets in ,but it does not used by the instance > nwfilter. > the instance nwfilter only has no-mac-spoofing > ,no-arp-spoofing,no-ip-spoofing ,and allow-dhcp-server filter. > > if I misunderstand some thing ,please correct me ,thks . > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack Foundation] OpenStack Foundation
2012/2/4 Soren Hansen : > 2012/2/4 Soren Hansen : >> The schedule on http://wiki.openstack.org/Governance/Foundation says >> a revised (or final?) mission statement would have been posted last >> week. > Ah, since I wrote my e-mail yesterday, it seems that page has been > updated. Sorry. I also see that the "DRAFT FOR COMMENT" designation at the top of the Mission page was removed a couple of days ago[1]. Is that intentional? [1]: http://goo.gl/e8lHf -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack Foundation] OpenStack Foundation
2012/2/4 Soren Hansen : > The schedule on http://wiki.openstack.org/Governance/Foundation says a > revised (or final?) mission statement would have been posted last week. Ah, since I wrote my e-mail yesterday, it seems that page has been updated. Sorry. We were promised more communication. I think the absolute minimum would be announcing changes to the schedule on the mailing list. More info about what is actually going on would also be nice. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack Foundation] OpenStack Foundation
It is now Feb 4th. The has been no updates since Jan 5th. Not that I've noticed, at least. The schedule on http://wiki.openstack.org/Governance/Foundation says a revised (or final?) mission statement would have been posted last week. Also, a draft structure should have been posted this week? Can we still expect that? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Libvirt File Injection
2012/1/30 Dan Prince : > On Mon, Jan 30, 2012 at 1:05 PM, Brian Waldon > wrote: >> After implementing a working version of file injection on Libvirt, a >> good question was brought up on the merge prop: how should we handle >> a file injection failure? Injection could fail for several reasons: >> missing necessary libraries, unsupported image formats and bad >> permissions are just a few. There seem to be two clear paths forward: >> >> 1) Log an error, set the instance to ERROR, add an asynchronous fault to >> the instance in the db >> 2) Log a warning, move on with the boot process > My preference would be to log a warning and move on with the boot > process (#2). Or perhaps we could address this with some sort of async > error messages concept? If getting those files injected isn't critical to getting the machine up and running, you can use one of the many other ways to get data into your instances. If the API calls says to inject a file, and we know that failed, we should bail out. I certainly wouldn't want to pay for something that isn't going to work. Sure, most of the times when I've booted things on EC2 or on the Rackspace cloud, I've done so interactively and have had the chance to see any errors, but that's going to be more and more of a niche use case. I fully expect the majority of consumer of OpenStack in the future to be applications wanting more ressources. If we can't deliver on their requests, we should fail early so that they can take appropriate action ASAP. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Libvirt File Injection
2012/1/30 Brian Waldon : > After implementing a working version of file injection on Libvirt, a good > question was brought up on the merge prop: how should we handle a file > injection failure? Injection could fail for several reasons: missing > necessary libraries, unsupported image formats and bad permissions are just > a few. There seem to be two clear paths forward: > > 1) Log an error, set the instance to ERROR, add an asynchronous fault to the > instance in the db > 2) Log a warning, move on with the boot process > > It's not obvious which of these is the best route to take from a user's > point of view. I'm currently leaning towards option 1 as I wouldn't want to > have an instance come up (and be billed for it) while it wasn't what I > explicitly requested. I agree with this. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Nova] Essex dead wood cutting
2012/1/31 Thierry Carrez : >> Given that hyper-v appears to be supported by libvirt, wouldn't it be >> ok to drop the direct support in return for that? I've taken a look >> at the spawn() code and I can't immediately see anything which would >> stop that from working. [...] > I prefer to remove it rather than shipping it broken (or in an unknown > state). I agree. If we were to accept it back, it would have to be accompanied by tests that we can actually run without HyperV being present. Preferably along the lines of the fake libvirt we have which exposes an interface identical to libvirt's, but only does book keeping. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] What's the web server used in the Nova API server?
It's not like we wrote our own webserver :) Eventlet has a wsgi container. We just went ahead and used it. Not only does it perform very well, it's also very straightforward to use for testing (compared to having to install and configure Apache to get going).. 2012/1/16 Michael Basnight : > Just curious, whats the reason we went with rolling our own instead of using > something like nginx/apache2/etc w/ mod_wsgi? > > On Jan 16, 2012, at 2:14 AM, Thierry Carrez wrote: > >> Joe Smithian wrote: >>> I browsed the openStack documentation but couldn't find information >>> about the Nova API server. >>> What's the web server used in the Nova API server? >>> Can we use a different web server such as Apache or Tomcat? >>> >>> I'd appreciate your comments. >> >> Nova uses Python eventlet WSGI servers. >> You can't directly use a different web server, though you can certainly >> place Nova API servers behind some other server. >> >> -- >> Thierry Carrez (ttx) >> Release Manager, OpenStack >> >> ___ >> Mailing list: https://launchpad.net/~openstack >> Post to : openstack@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp > > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack Foundation] OpenStack Foundation
2012/1/5 : > To kick things off I've created a Foundation page on the wiki and published a > "Foundation Mission" draft for comment as well a a rough timeline for the > next couple of months: http://wiki.openstack.org/Governance/Foundation I've always imagined the foundation would be responsible for ensuring that developers will have access to a set of reference platforms for testing as well as ensuring that we have relatively continuous access to means for conducting large-scale testing. Can you weave that in there somehow? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Update on integration test gating
2012/1/5 James E. Blair : > --- > Loading install_requires with the contents of pip-requires > isn't getting us any real beneift and is causing issues. > > a) It can conflict with installing nova into an environment > where deps have been installed from packages (devstack) pip is well aware of system installed libraries if you're not using a virtualenv with --no-site-packages. This probably doesn't change anything really, I just wanted to mention it. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Who runs lists.openstack.org?
Who runs lists.openstack.org? Specifically, who can look into fixing the archives? I'd be happy to help out, I don't have a login on the box. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack Foundation] OpenStack Foundation
2012/1/5 Jim Curry : > So now for the update. In reality, there has been a ton of work > occurring to build a proposal that can serve as the basis for a > constructive community conversation. I firmly believe that the best > way to have a conversation with a group as diverse and large as the > OpenStack community is to start with a very defined proposal that > everyone can react to — positively or negatively. This proposal is > not meant to be anything other than a framework to manage the > discussion. No decisions are being made without the involvement of > the broad community. What does that mean? From what I understand so far, someone at Rackspace will eventually publish some documents for discussion and will "listen to feedback". What if this feedback isn't unanimous? Are shooting for a minimum that we can reach consensus on (by ripping out everything that anyone at all disagrees with) or will there be a vote on individual points of contention? Will the documents as a whole be put to a vote? If so, what if they're rejected? In both cases, who gets a vote? > Trying to handle the large number of complex issues over a mailing > list without a base framework would be impossible to manage. It is > also unfair to community members who don't live on the mailing lists > day to day, but do want to review the plans and provide comments. I'm much more concerned about the quality of the feedback than the quantity. Twitter only lets you make a point in 140 characters. That's hardly enough for a delicate discussion like this. A webinar isn't any better, because these things need carefully thought out arguments, not just whatever you can come up right there on the spot. Honestly, if people have strong opinions on this subject, but can't be bothered to send an e-mail, that's sad. > We are basing this initial proposal on a lot of input received from the > community and beyond — developers, users, companies, other open source > projects and foundations, lawyers, specific country experts, etc. How has this worked? Have these people (who?) been asked a specific set of questions? Can we see them so that we can consider the same questions? Can we see the feedback itself? If you expect the foundation after it's been established to take it into consideration at all, it needs to be disclosed, otherwise the first revision of the bylaws/constitution might go against this feedback entirely, and then it's been a waste of time. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Problems with run_tests.sh on 11.10
2012/1/2 Monty Taylor : > Do you think someone would be willing to accept that patch? Or should we > make our own branch with the patch applied and reference that in the > pip-requires? If this is something we want to use for anything half serious, I think we should create a branch of our own (or pester upstream hard enough that they finally get around to fixing it). -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Metadata and File Injection
2012/1/2 Ewan Mellor : >> > In the context of this discussion, that means that OpenStack needs >> > to work in DHCP-free environments, because we already know of many. >> The only one I'm familiar with is Rackspace where I think (and please >> do correct me if I'm wrong) DHCP isn't used only because Windows >> instances can't be configured with DHCP if they have multiple >> interfaces. > DHCP may be forbidden because: > * it has been deemed less reliable than pushing fixed IP addresses > from an asset database. If you already have a comprehensive asset > database, why push the addresses to a DHCP server to have it push them > to your hosts when you can push them to the host directly? Because Nova isn't likely to interface with your asset database. It maintains has its own database for these things. Also, how do you push something from an asset database, if you don't have network connectivity? > * it has been deemed less reliable than pushing addresses from the > virt layer. By whom and for what reasons? Every single operating system of any relevance at all understands DHCP. It's an extremely widely used system. It may not be perfect, but that will hold true for any replacement we decide to build. You'll simply be replacing the bugs, problems and idiosyncracies with a different set of bugs, problems and idiosyncracies. > Why go to the effort of making a DHCP service highly available, and > making sure that the failover is quicker than the timeout that causes > guests to decide to take zeroconf addresses instead? Just set the lease time to a couple of weeks. If you can't manage to fix a dead dnsmasq within that timeframe, that's probably the least of your problems. > * it has been deemed a security risk. Compromising the DHCP server > gives you all sorts of easy attacks. The DHCP server runs on the network controller. If your network controller is compromised, you're pretty screwed anyway, afaics. > This one actually seems like a very reasonable concern in the case of > tenant networks in a cloud. Pushing the address from the virt layer > takes away that attack vector altogether. ...and replaces it with another. > * the guy who is installing the cloud doesn't control the network, and > so the DHCP server isn't his and/or he's not allowed to run one of his > own. This seems likely in an academic environment or pilots within an > enterprise. a) This isn't a reason. It's just repeating the original non-problem: "it's not allowed". b) There are several ways in which we can ensure that our DHCP stuff doesn't bleed into the rest of the network. dnsmasq already won't respond to requests from mac addresses that it doesn't know about, but if that isn't good enough (why would this be? what's so darned special about dhcp requests anyway?), we can add extra strict filtering in a couple of places. > Note that none of these has to be specifically a decision about > OpenStack or clouds in general. They might just be blanket policies > from simpler times. That doesn't make them any easier to change. I can't tell you how to run your business. All I know is that if a client of mine gave me a functional requirement specification that would be perfectly met by DHCP, but they had a piece of paper from the mid-90's that said "DHCP isn't allowed. Just because." on which they refused to budge, I'd pass on the contract. I personally don't want to facilitate stupid policies. That's my policy. > Regardless of those four, your one seems like a complete deal-breaker > to me. If that's true (and it's not something that I'm aware of one > way or another) then we shouldn't even be having this discussion. We > can't build a solution that can't reliably configure Windows VMs. a) I don't think we should generally limit ourselves to whatever Windows can support. (Much like how I don't think we should have a hard dependency on hypervisors that have a Xenstore-like backdoor into the guest) b) I don't think I've heard any reason why we couldn't just configure one of the NIC's over DHCP, and then fetch the remaining network config from a network-connected metadata service. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Problems with run_tests.sh on 11.10
2012/1/2 Lorin Hochstein : > Would this work for Fedora-based distributions as well? Yes. The source checkout will include a debian/ directory, but that won't affect the Python code installed. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Metadata and File Injection
2012/1/2 Ewan Mellor : >> I disagree. I think it's very important indeed whether the reasons are >> good or stupid. If they're good, they'll probably apply to others as >> well, so we need to know about them, understand them, and take >> appropriate action. If the reasons are stupid, whatever solution we >> come up with might be deemed forbidden as well for similarly stupid >> reasons. The only winning move is not to play. > I didn't mean that we shouldn't try to understand the reasons. I > meant that it doesn't matter whether they are good reasons or bad ones > because either way the policy is there, and we're not going to change > it. If you want to sell to that customer then you have to work with > their environment. I understand. My point is just that if we don't understand (or are even aware of) the reasons there's no way for us to know that whatever solution we come up with won't trigger the same (silly or reasonable) rules. There's no way I want to waste time (others are free to do with their time whatever they please, of course) playing some silly game where a company has decided (for no apparent, good reason) to disallow DHCP, and we come up with some amputated, under-engineered, not-tested-in-the-real-world-at-all alternative that just happens to not be called DHCP, and then watch them disallow that, because they don't like the colour of the box it came in, and then we scramble to develop yet another functionally equivalent, but differently named dynamic host configuration protocol, etc. etc. etc. The only way to win that game is to refuse to play. If there are appropriately articulated, sound reasons to avoid DHCP, let's make them heard so that we can address them. "We don't allow DHCP. Period." isn't even a reason, and if that were all we had to go on, we could just call it something else, but let it accidentally be compatible with DHCP. The magic of ridiculous rules is that you can come up with equally ridiculous workarounds. > In the context of this discussion, that means that OpenStack needs to > work in DHCP-free environments, because we already know of many. The only one I'm familiar with is Rackspace where I think (and please do correct me if I'm wrong) DHCP isn't used only because Windows instances can't be configured with DHCP if they have multiple interfaces. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Configure Rate limits on OS API
2012/1/2 Jay Pipes : > On Mon, Jan 2, 2012 at 6:27 AM, Soren Hansen wrote: >> 2011/12/29 Vishvananda Ishaya : >> I can't think of any reason we can't just fork() N times (where N >> defaults to number of cores, but could be overridden by a flag) after >> socket.listen(), but before we go into the socket.accept() loop? > > This is precisely what Swift does. And, very shortly, Glance: > > https://review.openstack.org/#change,2486 Ah, neat! Now, if only we could get stuff like this moved into openstack-common :) -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Openstack Nova Prunning/ Consistency mecanisms
2011/12/29 Razique Mahroua > # Consistency > - network consistency > If I have a /24 networks, that means I will have 256 fixed_ips, is it > possible that nova make sure there are 256 ips for that network I don't understand what you mean here. How could there /not/ be 256 IP's in a /24? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Integration test gating on trunk
2011/12/29 James E. Blair : > Having said that, the Jenkins job has been running in silent mode on > master for several days with few false errors. My feeling from the > design summit was that it was generally understood there would be a > shakedown period, and people are willing to accept some risk and some > extra work for the benefits an integration test gating job will brink. > I think we're at that point, so I'd like to turn this job on Tuesday, > January 3rd. I'm much happier with a test rig that has the occasional false negative than no testing at all. A big +1 from me. Thanks for this excellent effort! -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Problems with run_tests.sh on 11.10
2011/12/30 John Griffith : > Oops, sorry about that. Forgot to check it in the venv, which reveals the > issue: > > % tools/with_venv.sh > jdg@grumpy ~/Projects/OpenStack/nova > % python > Python 2.7.2+ (default, Oct 4 2011, 20:06:09) > [GCC 4.6.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> import M2Crypto > Traceback (most recent call last): > File "", line 1, in > File "/usr/local/lib/python2.7/dist-packages/M2Crypto/__init__.py", line > 22, in > import __m2crypto > ImportError: /usr/local/lib/python2.7/dist-packages/M2Crypto/__m2crypto.so: > undefined symbol: SSLv2_method >>>> Ah, yes. That's because M2Crypto hasn't kept up wit the removal of SSLv2 from OpenSSL. It's fixed in the Ubuntu packages, so if you remove the M2Crypto line from pip-requires and put this instead: -e bzr+http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/precise/m2crypto/precise/#egg=M2Crypto You should be fine. (Yes, the line in pip-requires should start with "-e") -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Configure Rate limits on OS API
2011/12/29 Vishvananda Ishaya : > In addition to the rate limiting changes, you can also run multiple copies > of nova-api on different ports and put a load balancer (like ha-proxy) in > front. You may have to set up your proxy to add an X-Forwarded-For header > and specify the > --use_forwarded_for > flag to get things like metadata to work. I can't think of any reason we can't just fork() N times (where N defaults to number of cores, but could be overridden by a flag) after socket.listen(), but before we go into the socket.accept() loop? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Metadata and File Injection
2012/1/1 Ewan Mellor : > Rackspace aren't the only people to have problems with DHCP. I have > encountered a number of enterprises where DHCP is expressly forbidden inside > the datacenter (for good reasons or stupid ones, that's not important). I disagree. I think it's very important indeed whether the reasons are good or stupid. If they're good, they'll probably apply to others as well, so we need to know about them, understand them, and take appropriate action. If the reasons are stupid, whatever solution we come up with might be deemed forbidden as well for similarly stupid reasons. The only winning move is not to play. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance functional tests failing
2012/1/1 Ewan Mellor : > How has a keystone change managed to break Glance when we're pinning Keystone > at a specific version? Could you try blowing away your .venv directory and trying again? Perhaps there's a stale Keystone lying around in there. It won't get replaced, because it already fulfills the stated requirement (given by egg=keystone). Perhaps we should change pip-requires to read "egg=keystone-dev" or something. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Quantum OVS over StackOPS
Please share the exact error message from after installing openvswitch.
Best regards,
Soren Hansen
2011/12/21 Alisson Soares Limeira Pontes
> When i first had the problem i did not have it installed.
> But then i installed it
>
> root@nova-controller:~# ovs-ofctl -V
> ovs-ofctl (Open vSwitch) 1.3.0
> Compiled Dec 20 2011 18:08:01
> OpenFlow versions 0x1:0x1
>
> but the problem persists.
>
>
> 2011/12/21 Soren Hansen
>
>> 2011/12/21 Alisson Soares Limeira Pontes :
>> > I am configuring nova to work with Quantum.
>> > When aplying the configurations for the OVS plugin I face this error.
>> >
>> > root@nova-compute-1:~# python ovs_quantum_agent.py
>> ovs_quantum_plugin.ini
>> >
>> > Traceback (most recent call last):
>> > File "ovs_quantum_agent.py", line 297, in
>> > plugin = OVSQuantumAgent(integ_br)
>> > File "ovs_quantum_agent.py", line 175, in __init__
>> > self.setup_integration_br(integ_br)
>> > File "ovs_quantum_agent.py", line 188, in setup_integration_br
>> > self.int_br.remove_all_flows()
>> > File "ovs_quantum_agent.py", line 89, in remove_all_flows
>> > self.run_ofctl("del-flows", [])
>> > File "ovs_quantum_agent.py", line 86, in run_ofctl
>> > return self.run_cmd(full_args)
>> > File "ovs_quantum_agent.py", line 58, in run_cmd
>> > p = Popen(args, stdout=PIPE)
>> > File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
>> > errread, errwrite)
>> > File "/usr/lib/python2.6/subprocess.py", line 1139, in _execute_child
>> > raise child_exception
>> > OSError: [Errno 2] No such file or directory
>> >
>> > Anyone know why? The file /usr/lib/python2.6/subprocess.py does exists!
>> > All steps before this (according to the OVS README) went well.
>> > I am using Quantum 2012.1-e2-ubuntu2 and the StackOPS0.3.
>>
>> It's complaining that it can't find ovs-ofctl. Did you install
>> openvswitch?
>>
>> --
>> Soren Hansen| http://linux2go.dk/
>> Ubuntu Developer| http://www.ubuntu.com/
>> OpenStack Developer | http://www.openstack.org/
>>
>
>
>
> --
> Alisson Pontes
> __
> Network Technology Evolution Researcher
> CPqD - Center for Research and Development in Telecommunications
> Tel.: +55 19 3705-4996
> apon...@cpqd.com.br
> www.cpqd.com.br
>
>
>
>
--
Soren Hansen| http://linux2go.dk/
Ubuntu Developer| http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Quantum OVS over StackOPS
2011/12/21 Alisson Soares Limeira Pontes :
> I am configuring nova to work with Quantum.
> When aplying the configurations for the OVS plugin I face this error.
>
> root@nova-compute-1:~# python ovs_quantum_agent.py ovs_quantum_plugin.ini
>
> Traceback (most recent call last):
> File "ovs_quantum_agent.py", line 297, in
> plugin = OVSQuantumAgent(integ_br)
> File "ovs_quantum_agent.py", line 175, in __init__
> self.setup_integration_br(integ_br)
> File "ovs_quantum_agent.py", line 188, in setup_integration_br
> self.int_br.remove_all_flows()
> File "ovs_quantum_agent.py", line 89, in remove_all_flows
> self.run_ofctl("del-flows", [])
> File "ovs_quantum_agent.py", line 86, in run_ofctl
> return self.run_cmd(full_args)
> File "ovs_quantum_agent.py", line 58, in run_cmd
> p = Popen(args, stdout=PIPE)
> File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
> errread, errwrite)
> File "/usr/lib/python2.6/subprocess.py", line 1139, in _execute_child
> raise child_exception
> OSError: [Errno 2] No such file or directory
>
> Anyone know why? The file /usr/lib/python2.6/subprocess.py does exists!
> All steps before this (according to the OVS README) went well.
> I am using Quantum 2012.1-e2-ubuntu2 and the StackOPS0.3.
It's complaining that it can't find ovs-ofctl. Did you install openvswitch?
--
Soren Hansen | http://linux2go.dk/
Ubuntu Developer | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] using objects returned from DB layer
2011/12/15 Jesse Andrews : > I agree except I though the preference was for > >> instance_uuid = instance['uuid'] > > not > >> instance_uuid = instance.uuid > > (use dict's and don't assume sqlalchemy) Yes... That's what Chris is saying, isn't it? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Subteam Changes
2011/12/11 Stefano Maffulli : >> Any time you find will be in someone's evening and in someone else's >> middle-of-the-night, and most of us probably also have busy calendars. > What is not clear to me is if you would like to reduce the amount of > meetings or just not add more. One step at a time :) > Which ones of the meetings listed on > http://wiki.openstack.org/Meetings/ are the most difficult for you to > attend to? I'm sorry that I can't attend the QA team meeting, the orchestration team meeting and the nova-db team meeting. I've accepted that my Tuesday evening is sacrificed on the altar of IRC meetings. After all, it's only the evening and only one of them. I feel quite sad that it's virtually impossible for anyone East of here to attend as it'll be during their night, though. >> My business hours are 9 - 17 (5 PM) CET. > In my experience many employers are happy to give brilliant and > experienced developers like you ample flexibility in terms of business > hours, offering compensation (monetary, free time, and other perks). > Are these hours your choice or are they mandated by your employer? I define my own working hours, but that's really beside the point. Calling from 2 AM - 3 AM "working hours" doesn't make it a convenient time slot, even if that means I take an hour off at a different time. No matter what you call it and how much time you take off at another time, it still means you have to get up in the middle of the night for a meeting. And if I have a meeting mid-evening, that evening is lost. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Subteam Changes
On Dec 10, 2011 1:13 AM, "Paul Voccio" wrote: > I'm not sure why this frustration is directed at me. Because you're talking about calling a meeting as being no big deal, which might be true if everyone were in the same office, but when we're spread across the planet, it *is* a big deal. > My point is sometimes you need to get people to make a decision and > waiting for a week for everyone to toss their .02 cents doesn't always > work. Now I'm confused. I thought you were arguing in favour of weekly meetings while I was arguing in favour of just raising subjects on the mailing list whenever you wanted. What are you arguing in favour of? > If the idea is to have subteams who care about small parts of code > then I should be able to go find those people to make a decision. Sure. And you can. We're right here on this very mailing list. In a project that is spread this far across the planet, you simply cannot expect to be able to just "call a meeting" and have everyone turn up. Any time you find will be in someone's evening and in someone else's middle-of-the-night, and most of us probably also have busy calendars. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Subteam Changes
2011/12/10 Trey Morris : > This problem has to have been solved before. Anyone know how > multinational companies deal with meetings? The distro team meetings at Canonical used to rotate 8 hours every week to distribute the pain. Eventually, though, we gave up having these big meetings altogether. > Also, Soren, to be fair, I'm don't believe I've heard anyone propose a > meeting that wasn't on ~utc-7 business time, yourself included. No, because I try not to propose meetings at all :) > Maybe there should be some give and take here? Perhaps this is where > give starts.. What time works for you? It certainly does seem fitting that the most pain falls upon the person proposing the meeting. Let's call it an incentive for people to not call meetings :) My business hours are 9 - 17 (5 PM) CET. In Central Time, that's 2 AM - 10 AM. In Pacific Time, that's 12 AM - 8 AM. In Japanese time, that's 17 - 1 AM. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Subteam Changes
2011/12/9 Paul Voccio : > I think the benefits of an "all hands" irc/irl meeting is to reduce > the overall amount of time needed to drive to a decision. I can > usually do this in a 30 minute meeting if I have the relevant people > and have a handful of items that need to be decided upon. Having to > have that same conversation with people usually takes a few days as > email is not usually treated as a real-time medium. Ok, let's assume that a decision will always be reached if a topic is on the agenda for a meeting. Let's also assume that points for discussion can pop up at any given time throughout the week. (For the record, I only believe that one of these assumptions is sound, but meh.) As such, with a weekly meeting, it can take anywhere between 0 and 168 hours (or an average of 84 hours) to reach a decision. I honestly don't see how that's better than a predictable few days. This project is bigger than your office. You simply can't expect to "have all the relevant people". Ever. How much Japanese attendance do we have at our meetings? It's exceedingly frustrating that this part of my objection is completely ignored: It will cost me an evening and will be in the middle of the night for Japan. You guys can spend each other's *working hours* on meetings all day long for all I care. This is (supposed to be) my spare time. We put *every* single meeting in this project in US business hours, *every* single meeting *outside* European and Japanese business hours, and *every* single design summit in the US and we're surprised there's a strong US bias? Seriously? Also, I stand by my other arguments against synchronous meetings and for e-mail discussions: >> Compared to IRC meetings: >> >> * No need for everyone to be in the same place at the same time. >> * No need for everyone to sit through a meeting where only a fraction >> of the topics are relevant to them. >> * Mailing list archives makes it simple to look up past discussions. >> * Much looser time limits. If I need to test a hypothesis before I make >> a statement, I can do that on my own time. That's very hard to do >> during a synchronous meeting. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Subteam Changes
2011/12/8 Vishvananda Ishaya : > I'm hesitant to do more meetings as well, but we do need some > coordination. I call false dichotomy on this. I don't buy the "without meetings, we can't coordinate anything" notion. > I will leave the one for Monday on the board for now. Lets have one > meeting with the goal being to never need another one. Ok. Let me know how it goes. :) As I said, I can't attend. > IDEA: We could have a "status" etherpad that everyone could keep > updated each week with current tasks, important blueprints, concerns > etc. What would that gain us compared to a status e-mail? The benefits of using e-mail instead of Etherpad: * They're easier to look up after the fact (mailing list arcvhies are already set up). * It enables dialogue about the contents. * Having to click the send button motivates you to actually finish it. Compared to IRC meetings: * No need for everyone to be in the same place at the same time. * No need for everyone to sit through a meeting where only a fraction of the topics are relevant to them. * Mailing list archives makes it simple to look up past discussions. * Much looser time limits. If I need to test a hypothesis before I make a statement, I can do that on my own time. That's very hard to do during a synchronous meeting. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Subteam Changes
2011/12/8 Thierry Carrez : > Soren Hansen wrote: >> 2011/12/7 Vishvananda Ishaya : >>> 1) Weekly meeting for team leads. This is a time for us to discuss blueprint >>> progress, multiple-team-related issues, etc. Going to shoot for Mondays at >>> 2100 for this one. I really need the subteam leads to commit to making this >>> meeting. We can discuss at the first meeting and decide if there is a better >>> time for this to occur. >> I have a conflict in that time slot every other week, and frankly, I'd >> really like to avoid more meetings. I'd by far prefer keeping to >> e-mail, and if we (over e-mail) find something that warrants more >> real-time discussion, we can put it on the agenda for our regular >> openstack team meeting on Tuesdays. > The problem is the general meeting has less and less time to dedicate to > each project, so Vish identified that he could not get efficient subteam > feedback and inter-subteam coordination from it... hence the need for a > new meeting slot. It could be a short one (30min) at start. I hear that, but I'd like us to at least *try* and get it to work without stealing yet another one of my precious evenings for another horribly inefficient meeting. There are lots and lots and lots of thing we can do to avoid meetings that we haven't even tried. As a start, team leads could send a weekly report about what the team has been up to, what the challenges have been, and what they'll be doing the next week. When I say "horribly inefficient meeting" I'm not implying that our meetings are more inefficient than anyone else's, but just that meetings in general are horribly inefficient. The amount of information per second is tremendously low, and everyone has to sit through the whole thing even if they only care about a tiny fraction of the agenda (if they're even so lucky to have a proper agenda). Can we try to identify what sort of information we want from the various teams and *then* figure out how to extract and distribute it rather than just throwing a meeting slot at it? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Subteam Changes
2011/12/7 Vishvananda Ishaya : > 1) Weekly meeting for team leads. This is a time for us to discuss blueprint > progress, multiple-team-related issues, etc. Going to shoot for Mondays at > 2100 for this one. I really need the subteam leads to commit to making this > meeting. We can discuss at the first meeting and decide if there is a better > time for this to occur. I have a conflict in that time slot every other week, and frankly, I'd really like to avoid more meetings. I'd by far prefer keeping to e-mail, and if we (over e-mail) find something that warrants more real-time discussion, we can put it on the agenda for our regular openstack team meeting on Tuesdays. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [GLANCE] Proposal: Combine the "container_format" and "disk_format" fields in 2.0 Images API
2011/12/6 Jay Pipes : > On Fri, Dec 2, 2011 at 3:48 AM, Soren Hansen wrote: >> 2011/12/1 Jay Pipes : >> There are basically two things that are relevant: The image type and the >> container format. >> >> The image type can be either of kernel, ramdisk, filesystem, iso9660, >> disk, or "other". > What value does "other" give the caller? It's meant to denote something that isn't a kernel, ramdisk, filesystem, iso9660, nor disk. Maybe swap space, maybe a raw block device for Oracle.. Something that's distinct from the other things, but isn't common enough to warrant its own designation. >> The container type can be: raw, cow, qcow, qcow2, vhd, vmdk, vdi or qed >> (and probably others I've forgotten). > What about OVA? As I understand it, OVA is the single, tar'd file > format that may store one or more virtual appliances that are in > formats like VHD. I consider that a transport format. Maybe my choice of nomenclature is off, but an OVA clearly (based on your description) holds a number of s which in turn holds (typically) disk images. I'd much rather if Glance would extract all the relevant information from the OVA, store the disk images (setting the appropriate type and format in the process) and then discard the OVA. Much like how we treat ami's. It's a transport format. > How does the following sound? Would this work for folks? > > type field, with the following possible values: > > * kernel > * filesystem > * ramdisk > * disk > * iso9660 Sure, I can live without the "other" type. > > format field, with the following possible values: > > * raw - This is an unstructured disk image format [...] > * qcow2 - A disk format supported by the QEMU emulator that can expand > dynamically and supports Copy on Write (Already responded about OVA). You're missing qed, qemu's next gen disk format. > Should there be another format value of: > > * iso - An archive format for the data contents of an optical disc (e.g. > CDROM). > > to correspond to the iso9660 type? No, "iso" isn't a format in the same sense as vmdk and qcow2, etc. > Or should images with the iso9660 type have a "raw" format value? Yes, your garden variety .iso is a raw formatted iso9660 filesystem. It could technically be converted to any of the other formats, but seeing as they're tightly packed (no need for sparseness) and read-only (no need for sparseness nor copy-on-write), there's rarely much gained from that (other then confusion). -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [GLANCE] Proposal: Combine the "container_format" and "disk_format" fields in 2.0 Images API
2011/12/5 Donal Lafferty : >> Perhaps the finer details of what "MIME-style categorization" is are lost on >> me. >> Can you elaborate? Your original example was "vhd/x-ms-tools" which, to my >> eye, is simply a container type string with a vendor part added. What am I >> missing? > 'vhd' isn't a container type. It's a disk format. See > http://glance.openstack.org/formats.html Err.. a) What on earth counts as a container type then? b) That's really beside the point. It's a "something" with a vendor part added. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [GLANCE] Proposal: Combine the "container_format" and "disk_format" fields in 2.0 Images API
2011/12/2 Donal Lafferty : >> I simply don't think adding a vendor part to the container type >> string is going to be a very good way to encode this. > Can this even be done with a MIME-style categorization? Perhaps the finer details of what "MIME-style categorization" is are lost on me. Can you elaborate? Your original example was "vhd/x-ms-tools" which, to my eye, is simply a container type string with a vendor part added. What am I missing? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Proposal for Lorin Hochstein to join nova-core
2011/11/29 Vishvananda Ishaya : > Lorin has been a great contributor to Nova for a long time and has been > participating heavily in reviews over the past couple of months. I think he > would be a great addition to nova-core. It's now been 6 days, there has been more than 5 +1's and noone has objected. Welcome to the team, Lorin. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Proposal for Mark McLoughlin to join nova-core
2011/11/29 Vishvananda Ishaya : > Mark is maintaining openstack for Fedora and has made some excellent > contributions to nova. He has also been very prolific with reviews lately. > Lets add him to core and make his reviews count towards potential merges! It's now been 6 days, there's been more than 5 +1's and noone has objected. Welcome to the team, Mark! -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] HPC with Openstack?
2011/12/4 Lorin Hochstein : > Some of the LXC-related issues we've run into: > > - The CPU affinity issue on LXC you mention. Running LXC with OpenStack, you > don't get proper "space sharing" out of the box, each instance actually sees > all of the available CPUs. It's possible to restrict this, but that > functionality doesn't seem to be exposed through libvirt, so it would have > to be implemented in nova. > - LXC doesn't currently support volume attachment through libvirt. We were > able to implement a workaround by invoking "lxc-attach" inside of OpenStack > instead (e.g., see > <https://github.com/usc-isi/nova/blob/hpc-testing/nova/virt/libvirt/connection.py#L482>. > But to be able to use lxc-attach, we had to upgrade the Linux kernel in > RHEL6.1 from 2.6.32 to 2.6.38. This kernel isn't supported by SGI, which > means that we aren't able to load the SGI numa-related kernel modules. Why not address these couple of issues in libvirt itself? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [GLANCE] Proposal: Combine the "container_format" and "disk_format" fields in 2.0 Images API
2011/12/2 Donal Lafferty : > The key in my email was to ask whether MIME-like specialisations were > appropriate either for combining characteristics of an image into a > single property. > > E.g. /. The example I provided was > / > > That second example came from observing that a VHD produced by > VHDTOOL.exe as posted on MSDN produced a file that could not be > understood by XenServer. In contrast, Ken Bell's 'DiscUtils' as > posted on Codeplex produced a VHD that worked fine. When I spoke to > Ken, he mentioned he'd noticed that VHDTOOL.exe generated a slightly > different format. Now, I doubt Microsoft would host a tool that > didn’t support their format. Therefore, there seems to be a > difference of opinion as to what constitutes a VHD. I understand there might be differences in implementations of the various formats. Sometimes, this is due to bugs (common if the format was reverse-engineered) or perhaps different (incompatible) versions of the same format. I don't think the correct way to encode these differences is making it "vendor" specific. As an example, vmdk's generated by QEmu are different from vmdk's generated by VirtualBox, and both of those are different that vmdk's generated by VMWare (which again generates different vmdk's depending on its version), but the compatibility matrix is complicated. I think all vmdk's from QEMu will work in VirtualBox and VMWare, but VMWare and VirtualBox can certainly both generate vmdk's that QEMu doesn't understand. Some of these differences are due to different versions of the vmdk format being used, and some are due to incomplete implementations of the formats. I simply don't think adding a vendor part to the container type string is going to be a very good way to encode this. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [GLANCE] Proposal: Combine the "container_format" and "disk_format" fields in 2.0 Images API
2011/12/2 Donal Lafferty : > During October I noticed that Microsoft's vhdtool.exe creates VHDs that > XenServer can't understand. Boy was that painful. > The underlying problem is that some vhd's should be described as VM specific. Can you elaborate on this, please? I don't think I understand what "VM specific" means. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [GLANCE] Proposal: Combine the "container_format" and "disk_format" fields in 2.0 Images API
2011/12/1 Jay Pipes : > structure tar'd up. However, I think this can be more easily > accomplished by consolidating the disk and container formats in the > 2.0 API to just a single format field with the possible values: > > ova - This indicates the data stored in Glance is an OVF container > that may actually contain multiple virtual appliances that has been > tar'd into the single-file OVA format > raw - This is an unstructured disk image format > vhd - This is the VHD disk format, a common disk format used by > virtual machine monitors from VMWare, Xen, Microsoft, VirtualBox, and > others > vmdk - Another common disk format supported by many common virtual > machine monitors > vdi - A disk format supported by VirtualBox virtual machine > monitor and the QEMU emulator > iso - An archive format for the data contents of an optical disc > (e.g. CDROM). > qcow2 - A disk format supported by the QEMU emulator that can > expand dynamically and supports Copy on Write > aki - This indicates what is stored in Glance is an Amazon kernel image > ari - This indicates what is stored in Glance is an Amazon ramdisk image > ami - This indicates what is stored in Glance is an Amazon machine image > > What do people think of this proposal to combine the two into a single > "format" field? I agree the current "disk_format"/"container_format" tuple isn't ideal. There's overlap between the two and at the same time, there are things that can't be expressed with the current selection of valid settings. I do think having two separate fields defining the contents, though. There are basically two things that are relevant: The image type and the container format. The image type can be either of kernel, ramdisk, filesystem, iso9660, disk, or "other". The container type can be: raw, cow, qcow, qcow2, vhd, vmdk, vdi or qed (and probably others I've forgotten). Container type is essential in deciding whether the hypervisor in question will be able to take the image and read its contents (i.e. map a block of data in the container to a block of data in the contained image). Image type is essential in deciding what to do with it. I.e. *don't* try to attach a kernel as a filesystem, *don't* try to use an iso9660 image as your kernel, *do* attach iso9660 images as CD's, not as hard drives, *do* accept booting a VM with only a disk image attached, *do* require a kernel if you have a filesystem image rather than a disk image, etc. At the moment, we try to guess the user's intent (if they don't pass a kernel, we just boot the image and hope for the best). This is error prone. "aki", "ari", and "ami" have always struck me as odd. If you upload an "aki" to OpenStack, by the time it actually reaches Glance, it's not an aki anymore. Its image type is "kernel" and its container format is "raw". It's indistinguishable from a raw kernel image uploaded by some other mechanism. Same for ari (ramdisk/raw) and ami (filesystem/raw). If anything, aki/ari/ami might be considered a (single) transport format. Uploading an image to EC2 involves a bundling process where the image in question is split up, signed (and encrypted?), uploaded to S3 along with a manifest and then "registered". Upon registration, the signature is verified, the image is decrypted(?), and stitched back together to form a kernel image (or ramdisk or machine image). At this point, any remnants of the manifest and the rest of the bundle are gone. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] reviewday
Can you explain what some of the errors mean? For instance, for this merge proposal: https://review.openstack.org/#change,1950 the unit tests passed, but the libvirt tests were "Hit by Torpedo" and XenServer tests had a "Chef client timeout". Am I right to assume that at least the chef client timeout problem is an issue in the test setup rather than indicative of a problem with the merge proposal? I have no idea what "Hit by Torpedo" means :) -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Providing packages for stable releases of OpenStack
2011/11/30 Mark McLoughlin : > On Wed, 2011-11-30 at 13:07 +0100, Soren Hansen wrote: >> I propose we start building packages from the stable branches and put >> them in an appropriately named/labeled PPA, such as >> nova-core/diablo-qa or nova-core/diablo-not-for-production (or >> perhaps under openstack-stable-maint). > I'm not convinced that distribution specific packaging is the best way > to go about this. That's a valid discussion. At the moment, this is what we do for trunk commits. This is how we generally propose that people test things out. I don't see any reason why the mechanics for testing the stable branches should be different. So, a discussion about these mechanisms shouldn't be isolated to the context of the stable branch. > I want Fedora users to be able to test out, and get involved with, > upstream as easily as Ubuntu users are. I'd be happy for us to build Fedora packages as well, fwiw. > Same for other distros. The thought of getting into the game of > maintaining this upstream packaging for multiple distros, and e.g. > having to make sure any dependencies are packaged for these distros > ... ugh. Yes, this is a lot of work. This is one of the primary reasons we chose a reference platform to begin with: Being able to focus the efforts and actually succeed rather than trying to do everything and fail. We had (and have) people involved in the project that could actually take this on. If someone wants to do the same for Fedora (and other distros), that'd be awesome. > I don't have anything concrete to offer as an alternative, but I'd > love to see something like devstack that runs either from git or > tarballs and supports multiple distributions. For production, we recommend people use packages. I think there's a lot of value in using the same installation mechanism for QA as for production. > There's also the likes of jhbuild, GARGNOME, minuteman and surely more > - perhaps we can take a leaf out of their books? I hope I'n not stepping on anyone's toes, but I consider those things to be relics from a time before things such as PPA's became prevalent. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Providing packages for stable releases of OpenStack
2011/11/30 Thierry Carrez : > Soren Hansen wrote: >> I propose we start building packages from the stable branches and put >> them in an appropriately named/labeled PPA, such as nova-core/diablo-qa >> or nova-core/diablo-not-for-production (or perhaps under >> openstack-stable-maint). >> [...] > That would work (and inside the current project). Just two remarks: > > * Expectations are difficult to control. Even if we use an intimidating > name, some people will still expect this to provide more than it > actually does. For example, people kept thinking that the "2011.3" > release PPA would be updated, while it explicitly said it wouldn't. The reason I want the PPA name to be scary looking is exactly because of the lesson learned from those PPA's. It's easy to miss the disclaimers on Launchpad (and if you happen to find the PPA info somewhere else, there might be no disclaimer at all!). The PPA name is the most obvious place to put this. Only if you're running someone else's script to enable it will you never see it. Some people will still miss it, but I think it's the best we can do. > * I don't think that's what Vish and Monty are after -- they > specifically mentioned the lack of a production-ready distribution > channel as the problem that we needed to solve Right. I agree we shouldn't do that. Someone else should. But I don't want that to hold back the creation of the per-commit PPA for diablo/stable which I find important for QA purposes. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Providing packages for stable releases of OpenStack
I think there are two distinct use cases here. To me, the PPA's have always been a QA tool. I wanted people willing to help test OpenStack to be able to do so with as little effort as possible. Building packages per-commit gave us that. It seems incredibly counterintuitive to me that someone who wants to help us verify the stable branches need to jump through more hoops to do so. IMO we should be as least as concerned about the quality of stable updates as anything else. This is why I think we should be offering a PPA with per-commit builds from the stable branch(es). This is completely different from a "production" PPA. I wouldn't dream of pointing people to the above mentioned PPA for their production environment. If someone wants to offer this outside of (but perhaps in cooperation with) OpenStack, that'd be great. I'd be delighted to see companies taking this on and offering a supported OpenStack distribution, but I don't think this is our job for pretty much all the same reasons Thierry outlines. I propose we start building packages from the stable branches and put them in an appropriately named/labeled PPA, such as nova-core/diablo-qa or nova-core/diablo-not-for-production (or perhaps under openstack-stable-maint). At the same time, I'd like to propose that we limit ourselves to fewer supported versions of Ubuntu (for trunk builds as well as these new, stable branch builds). Specifically: * Most recent LTS * Most recent release (which may or may not be an LTS) * Current development release LTS's would go out of support when the subsequent LTS's first point release is released. Non-LTS's would go out of support a month after the subsequent release is out. This means that right now, we'd build: * Lucid (until 12.04.1 is released (July 2012)) * Oneiric (until May 2012) * Precise (until (probably) July 2014) This gives people ample opportunity to upgrade to the next release and at the same time reduces the amount of releases we need to worry about significantly. I think we'd get a valuable QA tool back and we'd reduce the burden of maintaining the per-commit packages by having fewer distro versions to worry about. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Proposal for Mark McLoughlin to join nova-core
2011/11/29 Vishvananda Ishaya : > Mark is maintaining openstack for Fedora and has made some excellent > contributions to nova. He has also been very prolific with reviews lately. > Lets add him to core and make his reviews count towards potential merges! I'd be delighted to have Mark on the core team. +1 -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Database stuff
2011/11/29 Jay Pipes : > On Tue, Nov 29, 2011 at 3:43 PM, Soren Hansen wrote: >> 2011/11/29 Jay Pipes : >>>> Besides, we don't really use transactions. I could easily read the >>>> same data from two separate nodes, make different (irreconcilable) >>>> changes on both nodes, and write them back, and the last one to write >>>> simply wins. >>> Sure, but using a KV store doesn't solve this problem... >> >> I'm not suggesting it will. My point is simply that using a KV store >> wouldn't lose us anything in that respect. > I see your point. But then again, it comes down to whether we care > about referential integrity or transactional safety. ...and right now we have neither (by choice, not by limitations imposed by the data store). Would you not agree? > If we don't, then we're just building a distributed system that has > unreliable persistent storage built into it, and that, IMHO, is a > bigger problem than the as-yet-unproven assertions around scalability > of a relational database in a distributed system. (more below) Yes. This is what we have now. And it sucks. >>> As soon as someone can demonstrate the performance, scalability, and >>> robustness advantages of rewriting the data layer to use a >>> non-relational data store, I'm all ears. Until that point, I remain >>> unconvinced that the relational database is the source of major >>> bottlenecks. >> I understand that MySQL (and the other backends supported by >> SQLAlchemy, too) scales very well. Vertically. I doubt they'll be >> bottlenecks. Heck, they're even well-understood enough that people >> have built very decent HA setups using them. I just don't think >> they're a particularly good fit for a distributed system. You can >> have a highly available datastore all you want, but I'd sleep better >> knowing that our data is stored in a distributed system that is >> designed to handle network partitions well. > I guess I don't understand this. How do you sleep at night TODAY > knowing that the data Nova stores in its persistent storage is wide > open to referential integrity problems and transactional state > inconsistencies? Not very well at all. If I thought everything was in good shape, I woulnd't have bothered with all of this :) > What's the point of having a data store that "understands network > partitions" if we don't care enough to protect the integrity of the > data we're putting in the data store in the first place? :( None at all. I hope I haven't said anything to suggest otherwise. MySQL simply was not designed to be distributed. Generally speaking, if you do end up in a situation where there's been a network partition and your master is on one side and you have a slave on the other side, a couple of things can happen: 1. You can automatically promote the slave to master, thus letting both sides of the partition keep going. 2. You can leave the slave be and let the entire one side of the partition be in read-only mode. I think the usual case is 1, since MySQL HA setups are usually designed to handle the case where the master dies rather than handling network partitions. Would you agree with this assertion? If both have acted as master, what happens when the the network is joined again? Hell breaks loose, because MySQL wasn't designed for this sort of thing. Something like Riak, on the other hand, is designed to excel for exactly this sort of situation. It makes no attempt to handle these conflicts (unless you explicitly tell it to just let last write win). If there are conflicts, you get to handle it in your application in whatever way makes sense. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Database stuff
2011/11/29 Jay Pipes : > On Tue, Nov 29, 2011 at 2:58 PM, Soren Hansen wrote: >> 2011/11/29 Jay Pipes : >>> There's a very good reason this hasn't happened so far: handling >>> highly relational datasets with a non-relational data store is a bad >>> idea. In fact, I seem to remember that is exactly how Nova's data >>> store started out life (*cough* Redis *cough*) >> To be fair, we're only barely making use of this in our DB >> implementation. I don't think we do any foreign key checking at all, >> and deletes (because we don't actually delete anything, we just mark >> it as deleted) don't cascade, so there are all sort of ways in which >> our data store could be inconsistent. > Because the database schema isn't properly protecting against > referential integrity failures does not mean the relational database > store is a failure itself. I'm not suggesting it's a failure at all. >> Besides, we don't really use transactions. I could easily read the >> same data from two separate nodes, make different (irreconcilable) >> changes on both nodes, and write them back, and the last one to write >> simply wins. > Sure, but using a KV store doesn't solve this problem... I'm not suggesting it will. My point is simply that using a KV store wouldn't lose us anything in that respect. >> In short, it seems to me we're not really getting much out of having a >> relational data store? > We're getting out of it what we ask of it. We aren't using scoped > sessions properly, aren't using transactions properly, and we aren't > enforcing referential integrity. But those are choices we've made, not > some native deficiency in relational data stores. I didn't mean to suggest that that was the case at all. The point I'm trying (but failing, clearly) to make is that with the way we're using it, we're not reaping the usual benefits from it, and that we'd in fact not lose anything by using a KV store. > As soon as someone can demonstrate the performance, scalability, and > robustness advantages of rewriting the data layer to use a > non-relational data store, I'm all ears. Until that point, I remain > unconvinced that the relational database is the source of major > bottlenecks. I understand that MySQL (and the other backends supported by SQLAlchemy, too) scales very well. Vertically. I doubt they'll be bottlenecks. Heck, they're even well-understood enough that people have built very decent HA setups using them. I just don't think they're a particularly good fit for a distributed system. You can have a highly available datastore all you want, but I'd sleep better knowing that our data is stored in a distributed system that is designed to handle network partitions well. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
It's been a bit over a week since I started this thread. So far we've agreed that running the test suite is too slow, mostly because there are too many things in there that aren't unit tests. We've also discussed my fake db implementation at length. I think we've generally agreed that it isn't completely insane, so that's moving along nicely. Duncan has taken the first steps needed to split the test suite into unit tests and everything else: https://review.openstack.org/#change,1879 Just one more core +1 needed. Will someone beat me to it? Only time will tell :) Thanks, Duncan! Anything else around unit testing anyone wants to get into The Great Big Plan[tm]? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Database stuff
2011/11/29 Jay Pipes : > There's a very good reason this hasn't happened so far: handling > highly relational datasets with a non-relational data store is a bad > idea. In fact, I seem to remember that is exactly how Nova's data > store started out life (*cough* Redis *cough*) To be fair, we're only barely making use of this in our DB implementation. I don't think we do any foreign key checking at all, and deletes (because we don't actually delete anything, we just mark it as deleted) don't cascade, so there are all sort of ways in which our data store could be inconsistent. Besides, we don't really use transactions. I could easily read the same data from two separate nodes, make different (irreconcilable) changes on both nodes, and write them back, and the last one to write simply wins. In short, it seems to me we're not really getting much out of having a relational data store? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Database stuff
2011/11/29 Vishvananda Ishaya : > On Nov 29, 2011, at 11:16 AM, Soren Hansen wrote: >> The way I'd attack these expensive-if-done-one-at-a-time-but-dirt-cheap- >> if-done-as-one-big-query is to have a method in the generic layer that >> is taylored for this use case. E.g. >> >> def instances_get_all_for_network_with_fixed_ip_addresses(): >> retval = [] >> for inst in instance_get_all_by_network(): >> x = inst.copy() >> x['fixed_ip_addresses'] = [] >> for ip in fixed_ip_get_by_instance(inst['id']): >> x['fixed_ip_addresses'].append(ip['address']) >> retval.append(x) >> return x > > I think there are a couple of issues with this approach: > > 1) combinatorial explosion of queries. Every time we need an > additional joined field we will be adding a new method. This could get > out of hand if we aren't careful. This is true. However, the code needs to live somewhere, so to speak. We can either have separate methods for separate use cases or the logic to support the separate use cases will just need to exist as conditionals in more generic methods. I'm just concerned we'll refactor this and end up with something else that is just as SQLAlchemy specific as the old implementation, only with a different API style. It's hard (for me, at least) to predict how other drivers will be able to achieve good performance, so I'd be wary of going down a particular path just because I can envision how I'd implement it with the SQLAlchemy backend. > 2) interface clarity. > Sometimes an instance dict contains extra fields and other times it > doesn't. This is especially annoying if the resulting object is > passed through a queue or another method. Does the instance object > have an address or not? Do I need to explicitly request it or is > it already embedded? Indeed. We have this problem already, though. If I do an instance_get, a bunch of things will already be joined in and available for me to use. However, if I get hold of an Instance object from somewhere else (say, from a FloatingIp object: floating_ip_ref['fixed_ip']['instance']) if I attempt to access something that hasn't been eagerly joined, I'll get the aforementioned, infamous DetachedInstanceError. If we switch to this other style instead, I think we'll fairly quickly get used to the idea that we shouldn't expect anything but the bare essentials to be available (e.g. only the id's of related object, not the objects themselves (for the one-to-many use case), and to get related objects (in the many-to-one use case), you have to call a different method altogether, etc.), and we'll automatically be more careful about our use of the various objects. Or so the hypothesis goes. > These are probably surmountable obstacles if we want to go this route. > I just want to point out that it has its own drawbacks. Agreed. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Database stuff
2011/11/29 Chris Behrens : > e) sounds good as long as we don't remove the ability to joinload up > front. Sometimes we need to join. Sometimes we can be lazy. With > 'list instances with details', we need to get all instances from the > DB and join network information in a single DB query. Doing 1 + (n*x) > DB queries to support a 'nova list' will not be acceptable when it can > be done in 1 query today. That's a fair point. Does my proposal (addressed to Vish on this list a couple of minutes ago) solve this for you? In short, for the situations where you need stuff joined, we have a separate method for that. If a db driver can make this request more efficiently, that's great. If it can't, there's a slow, simple fallback. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Database stuff
2011/11/29 Vishvananda Ishaya : > e) is the right solution imho. The only reason joinedloads slipped in is for > efficiency reasons. > > In an ideal world the solution would be: > > 1) (explicitness) Every object or list of related objects is retrieved with > an explicit call: > instance = db.instance_get(id) > ifaces = db.interfaces_get_by_instance(id) > for iface in ifaces: > ip = db.fixed_ip_get_by_interface(iface['id']) > 2) (efficiency) Queries are perfectly efficient and all joins that will be > used are made at once. > So the above would be a single db query that joins all instances ifaces and > ips. The way I'd attack these expensive-if-done-one-at-a-time-but-dirt-cheap- if-done-as-one-big-query is to have a method in the generic layer that is taylored for this use case. E.g. def instances_get_all_for_network_with_fixed_ip_addresses(): retval = [] for inst in instance_get_all_by_network(): x = inst.copy() x['fixed_ip_addresses'] = [] for ip in fixed_ip_get_by_instance(inst['id']): x['fixed_ip_addresses'].append(ip['address']) retval.append(x) return x And then, in the sqlalchemy driver, I could override that method with one that issues a query with joinedloads and all the rest of it. The intent is explicit, drivers that have no speedier way to achieve this get a free implementation made up of the more primitive methods. fixed_ip_get_by_instace might also have a default implementation that issues a fixed_ip_get_all() and then filters the results. This way, a new driver would be quick to add, and then we could optimize each query as we move along. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Database stuff
Hi, guys.
Gosh, this turned out to be long. Sorry about that.
I'm adding some tests for the DB api, and I've stumbled across something
that we should probably discuss.
First of all, most (if not all) of the various *_create methods we have,
return quite amputated objects. Any attempt to access related objects
will fail with the much too familiar:
DetachedInstanceError: Parent instance is not
bound to a Session; lazy load operation of attribute 'other_things'
cannot proceed
Also, with the SQLAlchemy driver, this test would pass:
network = db.network_create(ctxt, {})
network = db.network_get(ctxt, network['id'])
instance = db.instance_create(ctxt, {})
self.assertEquals(len(network['virtual_interfaces']), 0)
db.virtual_interface_create(ctxt, {'network_id': network['id'],
'instance_id': instance['id']}
self.assertEquals(len(network['virtual_interfaces']), 0)
network = db.network_get(ctxt, network['id'])
self.assertEquals(len(network['virtual_interfaces']), 1)
I create a network, pull it out again (as per my comment above), verify
that it has no virtual_interfaces related to it, create a virtual
interface in this network, and check the network's virtual_interfaces
key and finds that it still has length 0. Reloading the network now
reveals the new virtual interface.
SQLAlchemy does support looking these things up on the fly. In fact,
AFAIK, this is its default behaviour. We just override it with
joinedload options, because we don't use scoped sessions.
My fake db driver looks stuff like this up on the fly (so the
assertEquals after the virtual_interface_create will fail with that db
driver).
So my question is this: Should this be
a) looked up on the fly,
b) looked up on first key access and then cached,
c) looked up when the parent object is loaded and then never again,
d) or up to the driver author?
Or should we do away with this stuff altogether? I.e. no more looking up
related objects by way of __getitem__ lookups, and instead only allow
lookups through db methods. So, instead of
network['virtual_interfaces'], you'd always do
db.virtual_interfaces_get_by_network(ctxt, network['id']). Let's call
this option e).
I'm pretty undecided myself. If we go with option e) it becomes clear to
consumers of the DB api when they're pulling out fresh stuff from the DB
and when they're reusing potentially old results. Explicit is better
than implicit, but it'll take quite a bit of work to change this.
If we go with one of options a) through d), my order of preference is
(from most to least preferred): a), d), c), b).
There's value in having a right way and a wrong way to do this. If it's
either-or, it makes testing (as in validation) more awkward. I'd say
it's always possible to do on-the-fly lookups. Overriding __getitem__
and fetching fresh results is pretty simple, and, as mentioned earlier,
I believe this is SQLAlchemy's default behaviour (somebody please
correct me if I'm wrong). Forcing an arbitrary ORM to replicate the
behaviour of b) and c) could be incredibly awkward, and c) is also
complicated because there might be reference loops involved. Also,
reviewing correct use of something where the need for reloads depends on
previous use of your db objects (which might itself be conditional or
happen in called methods) sounds like no fun at all. With d) it's
pretty straightforward: "Do you want to be sure to have fresh responses?
Then reload the object. Otherwise, behaviour is undefined." It's
straightforward to explain and review. Option e) is also easy to
explain and do reviews for, btw.
DB objects with options a) through d) will have trouble making it
through a serializer. After dict serialisation, the object isn't going
to change, unresolved related objects will not be resolved, and
prepopulating everything prior to serialisation is out of the question
due to the potential for loops and very big sets of related objects (and
their related objects and their related objects's related objects,
etc.). I think it would be valuable to not have to think a whole lot
about whether a particular db-like object is a "real" db object or
whether it came in as a JSON object or over the message bus or whatever.
Only option e) will give us that, AFAICS.
It seems I've talked myself into preferring option e). It's too much
work to do on my own, though, and it's going to be disruptive, so we
need to do it real soon. I think it'll be worth it, though.
--
Soren Hansen | http://linux2go.dk/
Ubuntu Developer | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Vulnerability Management concerns: negativity & count
2011/11/24 Lloyd Dewolf : > On Thu, Nov 24, 2011 at 8:03 AM, Soren Hansen > wrote: > 2011/11/24 Lloyd Dewolf : >>> A. As my former boss, as of this week, Matt Mullenweg [1] would so >>> often remind us, "don't be so negative" -- he literally reminded my >>> VIP Services sub-team of that last week -- it's natural when you are >>> deep in the trenches. Instead use "Words that Work". [2] >> This is not marketing material. It's not meant to sell anything or >> convince anyone of anything. It's supposed to accurately convey what >> this team is and what it isn't. If you want to rephrase it, knock >> yourself out, but being unambiguous trumps "sounding good". You don't >> see legislation being rephrased to make it sound better either :) > Hi Soren, Hello. > I may be misreading, but both your response and part of ttx's reads to > me as a straw man argument -- you give back a single unrelated phrase > as opposed to demonstrating the value of all three phrases. I have no idea what you're saying here? You complain that the text in question sounds negative. I point out that its purpose doesn't warrant putting lots of effort into phrasing it any differently just for the sake of it and give another example of a type of text where the same holds true. I then move on to suggest that if you think it's important, you go for it. I'm not sure what it is that you're referring to as "unrelated"? > I'm frustrating by your mention of "marketing material" and ttx's > posslbe fallback of "technical page". What is the context of that? The context? I'm again not sure what you're asking. As I said, the page does not exist to convince anyone of anything. It's not a piece of marketing or recruitment material. Its purpose is to explain what has been agreed. > If I were to guess where you are coming from, which I hate doing, my > response would good communication is accessible to many audiences, > encourages participation (is positive!), translates well (hard!), and > still meets the needs of us pendantic fools. To be blunt, I think that's a waste of time. As long as it accurately explains what it's meant to explain, it can be harsh, negative, humourous, sad, happy, prosaic, poetic, whatever. I don't care, as long as it's not at the expense of unambiguity. > Second, unambiguous? That doesn't ring true to me. One sentence, the > first sentence, is about what the list is, followed by a whole > paragraph on what it isn't? Maybe, let's start with fleshing out that > first paragraph. My point is that if you want to rephrase it because you find it too negative, that's fine, *as long as it doesn't negatively affect its accuracy/unambiguity*. If it's already ambiguous, please explain how, so that we can address it. > Three times a lady? [1] ?!? > I think there is an opportunity to be concise, eliminate the seeding > of fear of immaturity and unprofessionalism, (translate better), and > get on with focusing that OpenStack has dedicated, profession > participants. Great. Go for it. > Future-me will be proud that we have a robust solution (which I feel > like you guys are challenging me to brainstorm on) and that we've > never had a premature disclosure. We're not quite a point yet where I'd consider that last point any sort of success. To me, it's kind of like celebrating that the shuttle hasn't exploded yet when the spaceship is still on the launch pad. > How can we get your fantastic expertises humoring me by exploring > solutions rather than throwing down spike strips. Nothing is worse > than the new guy also offering "solutions" [3] when the relevant > issues have already been well considered, often multiple times, and > where the participants likely already have some other solutions that > might be voted up by the context of additional considerations. So far, this conversation hasn't been about the contents the wiki page in question. It's been about its language. I'm at least as much of a language nut as the next guy, but I don't get paid to be a copy writer. I can't justify spending time making sure our policy documents are phrased in a specific way. Don't get me wrong. I think there are lots and lots and lots of text under this project's purview where these concerns are paramount. That page just isn't it. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Vulnerability Management concerns: negativity & count
2011/11/24 Lloyd Dewolf : > A. As my former boss, as of this week, Matt Mullenweg [1] would so > often remind us, "don't be so negative" -- he literally reminded my > VIP Services sub-team of that last week -- it's natural when you are > deep in the trenches. Instead use "Words that Work". [2] This is not marketing material. It's not meant to sell anything or convince anyone of anything. It's supposed to accurately convey what this team is and what it isn't. If you want to rephrase it, knock yourself out, but being unambiguous trumps "sounding good". You don't see legislation being rephrased to make it sound better either :) -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Vulnerability Management concerns: negativity & count
2011/11/24 Thierry Carrez : > This is actually linked to the next section. If you limit the numbers of > members in a vulnerability handling team, you create resentment with > those members or companies that are not part of it. The phrasing is > there to reassure non-members that there is no advantage for being "in". Exactly. We're bootstrapping the team and the process. We (as a project) don't necessarily know the people stepping forward to take on a membership of this team, so it's important that the responsibilities (of which there are many) and privileges (of which there are really none) are clear. I see no reason not to be clear about the ground rules up front, and make it explicit that it's not an "early warning list". It's a response team. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
2011/11/24 Sandy Walsh : > haha ... worse email thread ever. > > I'll catch you on IRC ... we've diverged too far to make sense. For anyone interested, this conversation continued on IRC yesterday. You can read it at the very end of http://eavesdrop.openstack.org/irclogs/%23openstack-dev/%23openstack-dev.2011-11-23.log as well as the very beginning of http://eavesdrop.openstack.org/irclogs/%23openstack-dev/%23openstack-dev.2011-11-24.log My internet connection disappeared just as we were finishing our discussion. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
2011/11/23 Sandy Walsh :
> I understand what you're proposing, but I'm backtracking a little.
> (my kingdom for you and a whiteboard in the same room :)
Well, IRC would be a good start. :) I haven't seen you on IRC for days?
> I think that you could have a hybrid of your
> db.do_something("desired_return_value")
I may be reading too much into this, but this example suggests you're
not following me, to be honest.
db.instance_create is not a method I'm adding. It is an existing
method. It's the method you use to add an instance to the data store,
so it's not so much about passing it "desired return values". It's
about adding an instance to the database in the exactly same fashion
as production could would have done it, thus allowing subsequent calls
to instance_get (or any of the ~30 other methods that return one or
more Instance objects) to return it appropriately. And by
appropriately, I mean: with all the same attributes as one of the real
db drivers would have returned.
> to produce:
> self.sorens_mox.Set(nova.db, 'instance_get_by_uuid', {'name': 'this or that',
> 'instance_type_id': 42})
We have this functionality today. My example:
self.stubs.Set(nova.db, 'instance_get_by_uuid', fake_instance_get)
was copied from one of our existing tests.
--
Soren Hansen | http://linux2go.dk/
Ubuntu Developer | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
2011/11/23 Sandy Walsh : > :) yeah, you're completely misunderstanding me. Likewise! :) > So, you've made a much better StubOutWithMock() and slightly better > stubs.Set() by (essentially) ignoring the method parameter checks and just > focusing on the return type. No, no. Read my e-mail again. I don't want to do it that way either. I showed two examples of what I'd like to get rid of, followed by what I'd like to do instead. > Side note: > I don't view tests that permit > exercise_the_routine_that_will_eventually_do_an_instance_get() > calls to be unit tests ... they're integration tests and the source of all > this headache in the first place. I meant "eventually" as in "it'll probably do a bunch of other things, but also do an instance_get", not as in "some number of layers down, it'll do an instance_get". > A unit test should be > exercise_the_routine_that_will_directly_call_instance_get() > > Hopefully we're saying the same thing on this last point? Absolutely. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
2011/11/23 Sandy Walsh : > Thanks Soren, I see what you're doing now and it makes perfect sense. > It'll be a nice helper class. Cool. > My only snipe would be that mox is generic to any library and this > fake only gives the benefit to db operations. We have to remember > "It's a db operation, so I have to do this. It's another method call > so I need to do that" I think if it more like "for db, I don't need to concern myself with test doubles. There's still a bunch of other stuff where that's not true, but for db, it Just Works[tm]." > How much effort would it be to make it into a better/more generic mox library? I don't see how that would even be possible? I'm writing a complete db driver, backed by Python dicts rather than sqlalchemy+sqlite. I can't imagine how you'd build something that generally can expose an API and behaviour identical to an arbitrary module, which seems to be what you're suggesting. Ok, that's not entirely true. I can imagine injecting a proxy in front of the real DB driver, record its behaviour, and on subsequent test runs, I'd return canned responses, but I really wouldn't recommend something like that. It's great of getting some insight into how a particular module is used. You can use that information when writing stubs, mocks, fakes, whatever based on it, but I wouldn't rely on being able to just replay its traffic and have everything work. Am I completely misunderstanding you? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
2011/11/22 Sandy Walsh :
> I suspect the problem is coming in with our definition of "unit
> tests". I don't think a unit test should be calling out of the method
> being tested at all. So anything beyond stubbing out the methods
> within the method being tested seems like noise to me. What you're
> describing sounds more like integration tests.
If I'm testing a method that includes a call to the db api, the strategy
with which I choose to replace that call with a double does not change
whether the test is a unit test or not.
I'm simply replacing this:
def test_something(self):
self.mox.StubOutWithMock(db, 'instance_get')
db.instance_get(mox.IgnoreArg(), mox.IgnoreArg()
).AndReturn({'name': 'this or that',
'instance_type_id': 42})
exercise_the_routine_that_will_eventually_do_an_instance_get()
verify_that_the_system_is_now_in_the_desired_state()
or this:
def test_something(self):
def fake_instance_get(context, instance_uuid):
return {'name': 'this or that',
'instance_type_id': 42}
self.stubs.Set(nova.db, 'instance_get_by_uuid', fake_instance_get)
exercise_the_routine_that_will_eventually_do_an_instance_get()
verify_that_the_system_is_now_in_the_desired_state()
with this:
def test_something(self):
ctxt = _get_context()
db.instance_create(ctxt, {'name': 'this or that',
'instance_type_id': 42})
exercise_the_routine_that_will_eventually_do_an_instance_get()
verify_that_the_system_is_now_in_the_desired_state()
Not only is this -- to my eye -- much more readable, but because the
fake db driver has been proven (by the db test suite) to give responses
that are exactly like what the real db driver would return, we have
better confidence in the output of the test. E.g. if the real db driver
always sets a particular attribute to a particular default value, it's
remarkably easy to forget to follow suit in an ad-hoc mock, and it's
even easier to forget to update the countless ad-hoc mocks later on, if
such a new attribute is added. This may or may not affect the tested
code's behaviour, but if that was easy to see/predict, we wouldn't need
tests to begin with :)
Over the course of this thread, I've heard many people raise concerns
about whether we'd really be testing the fake or testing the thing that
depends on the fake. I just don't get that at all. Surely a fake DB
driver that is proven to be true to its real counterpart should make us
*more* sure that we're testing our code correctly than an ad-hoc mock
whose correctness is very difficult to verify?
> I thought the motive of your thread was to create
> fast/small/readable/non-brittle/maintainable tests.
The motive was to gather testing related goals, action items, thoughts,
complaints, whatever. It just so happens that a lot of people (myself
included) think that speeding up the test suite and categorising tests
into "true unit tests" and "everything else" are important things to
look at.
> Integration tests, while important, make this goal difficult.
I agree. I'm very happy that there's a lot of people doing a lot of work
on the integration test suite so that I can focus more on unit tests. As
I think I've mentioned before, unit tests are really all we can expect
people to run.
> So, if we're both talking about real unit tests, I don't seen the
> benefit of the fake.
Please elaborate (with my above comments in mind).
> As for my example of 123 vs "abc", that was a bad example. Let me
> rephrase ... in one test I may want to have an environment that has no
> pre-existing instances in the db. In another test I may want to have
> an environment with a hundred instances.
>
> I'd like to understand how configuring the fake for both of these
> scenarios will be any easier than just having a stub. It seems like an
> unnecessary abstraction.
First of all, the DB is blown away between each individual test, so we
don't have to worry about its initial state.
In the first scenario, I'd do nothing. I have a clean slate, so I'm good
to go. In the second scenario, I'd just do 100 calls to
db.instance_create. With the mock approach, I'd write a custom
instance_get with a 100 if/elif clauses, returning whatever makes sense
for the given instance_id. Mind you, the objects that I return from my
mock may or may not be valid Instance objects. I can only hope that they
are close enough.
--
Soren Hansen | http://linux2go.dk/
Ubuntu Developer | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
2011/11/22 Mark Washenberger : > I think we all agree that we have a mess with the database stubbing that is > going on. And I'm confident that the db fake would make that mess more > manageable. > But the way I see the mess, it comes from having a giant flat db interface > and really large testcases. Those are definitely problems, too. The lack of an alternative DB driver (be it a fake or a real one) hasn't helped, either :( > Also, by having such large test cases, the generality & complexity of setUp() > as well as its distance from any given test method increase tremendously. I agree completely. > All of this makes it harder to understand the impact of any one line in setUp > and nearly impossible to tell which parts of setUp are relevant to a given > test. Absolutely. At the moment, it's extremely difficult to tell which parts of the setUp is actually needed for each individual test. > But, I have to admit that I'm not backing up the way I think we could make > the best improvements with any code at the moment--and Soren is. So I'm not > sure I really want to stand in the way. :) I'm hoping the code and the changes it will bring along will speak for itself once I'm done with it. I've implemented 138 of 291 db api methods in my fake driver. I don't know if I'll need all 291 for the unit tests to pass. I hope so, but I'm not sure at all. > My only detraction from a big fake is that it in general has to be a pretty > faithful implementation of the real interface. Again, with my fake db driver, I'll submit a test suite for it. A test suite that we can run against *any* db driver (real, fake, whatever) to verify that they're all faithful. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
2011/11/22 Soren Hansen : >> (real) Unit tests are our documentation and having to jump around to find >> out how the plumbing works doesn't make for good documentation. > I agree. That's exactly why I *don't* want mocks (for this) in the unit tests. It's been pointed out to me that I'm not making sense here. Thanks, Devin. :) The point is that I don't want to deal with the plumbing *at all*. I just want to be able to stick stuff in the data store and then let my production (as in non-test) code be able to pull it out again without mucking about with mocks (har har). -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
