[Openstack] TC candidacy
I'd like to hereby nominate myself as candidate for a seat on the Technical Committee. I'm a Senior Software Engineer at Cisco Systems. In the past, I've held similar positions at Nebula, Rackspace and Canonical. I've been part of this project since before it was called OpenStack, mostly focusing on Nova, but also spending a lot of time building up a lot of the automation (including the CI infrastructure) to help us hit the ground running in the very beginning. Monty's team is doing a fantastic job looking after the infrastructure stuff now, so I've almost entirely backed off from that. I'm a core developer of Ubuntu and hold a seat on Ubuntu's technical board. In the course of my work on Ubuntu, I've contributed to countless open source projects, in recent years mostly focusing on virtualisation related things. I think our most important qualities in OpenStack are reliability and scalability and this will very likely shine through in my work on the technical committee. Even though I've spent most of my OpenStack time working on Nova, I have a keen sense of the bigger picture, both in terms of considering the other OpenStack projects, but also considering our various upstreams and downstreams. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OSSA 2012-014] Revoking a role does not affect existing tokens (CVE-2012-4413)
So if I can grant people access to a particular tenant, I can invalidate everyone's tokens at will now? Best regards, Soren. Sent from my phone. Please pardon my brevity. On Sep 12, 2012 6:40 PM, Thierry Carrez thie...@openstack.org wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 OpenStack Security Advisory: 2012-014 CVE: CVE-2012-4413 Date: September 12, 2012 Title: Revoking a role does not affect existing tokens Impact: High Reporter: Dolph Mathews (Rackspace) Products: Keystone Affects: Essex, Folsom Description: Dolph Mathews reported a vulnerability in Keystone. Granting and revoking roles from a user is not reflected upon token validation for pre-existing tokens. Pre-existing tokens continue to be valid for the original set of roles for the remainder of the token's lifespan, or until explicitly invalidated. This fix invalidates all tokens held by a user upon role grant/revoke to circumvent the issue. Folsom fix: http://github.com/openstack/keystone/commit/efb6b3fca0ba0ad768b3e803a324043095d326e2 Essex fix: http://github.com/openstack/keystone/commit/58ac6691a21675be9e2ffb0f84a05fc3cd4d2e2e References: https://bugs.launchpad.net/keystone/+bug/1041396 http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=2012-4413 Notes: This fix will be included in the future Keystone 2012.1.3 stable update and the upcoming Folsom-RC1 development milestone. - -- Thierry Carrez (ttx) OpenStack Vulnerability Management Team -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQIcBAEBCAAGBQJQULoUAAoJEFB6+JAlsQQjGacQAJUvJb+oIjh73KAYYuDpl/YP PqJa4nmjVin7CyQ8AbxHK63xrAQ7isPFpCCqtEmjZ5kvFCrJRHiQggHNqISRhnvo +HyS6RSn4Vrp001PSZSmQI5MpgkeWhbOy+fk4/ZY7hFgUyS2YqC8YiK7DTMdKRBi toWOHRVWrmA4fUEDDcDdm9XzRseTC0cZAbj9bYAF+vXPdpxeGpq5l9Kb6yDezXGD 62dFvHghVTWdUIN+gK4V4d77PoyeO9NRd4Ud0GjDpV/asQL31dW6B4aRPYVDPhL3 7xcnhRsnZ3Y5J31n+7E/gMF+J+6kOaY/DNFZQ8chNW18kplYnmJnm7s3BJNjD512 UF/S5A5sH1Rk/vwe2nAHSqvQ1Dq3K0sRvW3YCijG2Rdj3mhBOr6OlvT5uJmnkeJT GQQ8SR3y+ZLS/2EEW+cVjDMxV4Gnf9Zzrw/tSjVp6QLmJAkG8qrFmgdisQ/Jao4M ygE8ZVu8lJq7N8b+k8XkB+bhz9E9V6hYOUuGoifEHRIPki/Ed7++BcdVTQdQYpAL kDTaoVZt1+plwAu4ZBLxUg1vhVz19qgDc7UeoY1sPc1JcRWp/ONnp6K4z+Y+7Rsx 3E4FLH0/qgFxKDHdGX91Plehk9dIEjHcGtKaXI8vOvGT17srYQaF6Y7rc+9TwaqI bggBCxcI2PLQgjuWyF4M =+6UN -END PGP SIGNATURE- ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack Foundation] Individual Nominations for Foundation Board of Directors
Sorry if it's already been made clear somewhere, but I'm unsure for what term these individuals are being elected? The bylaws say that the election happens each year in the first two weeks of January, so is this an election to fill the seats until the first ordinary election in January? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack Foundation] Individual Nominations for Foundation Board of Directors
Nope, that's perfectly clear, I just failed to spot it. Thanks. Best regards, Soren. Sent from my phone. Please pardon my brevity. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova] [cinder] Nova-volume vs. Cinder in Folsom
Apologies if this has already been proposed, but how about an option 3 (perhaps more accurately option 2.5): We already have a process for maintaining code in one project and occasionally copying it to another project. Namely, code is maintained in openstack-common and then -- at appropriate times -- gets copied to Nova or Glance or whatever. Correct? Seeing as Cinder is supposedly a straight copy of Nova volume, it seems feasible to occasionally copy it all back into Nova. This way, it's not a matter of fixing bugs (and adding features and whatever) twice, but rather fixing bugs (and adding features and whatever) once and the rest is straight-forward (possibly even easily scriptable) patch management. Obviously, this wouldn't happen indefinitely, but simply serve to bridge the gap between those who want to split it out (with which I can certainly sympathise) and those who want to keep it Nova for Folsom (which I can also sort of relate to). -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Greatest deployment?
2012/5/30 Matt Joyce matt.jo...@cloudscaling.com: Secondly, while LXC does provide a lot of native access, it still does paging management internally just as kvm does. So direct memory management ( some HPC users like this ) becomes just as problematic as it is in kvm. Lots of overhead. I'm not convinced this is accurate. Can you provide some kind of reference for this? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Greatest deployment?
2012/5/30 Matt Joyce matt.jo...@cloudscaling.com: On Wed, May 30, 2012 at 12:39 PM, Soren Hansen so...@linux2go.dk wrote: 2012/5/30 Matt Joyce matt.jo...@cloudscaling.com: Secondly, while LXC does provide a lot of native access, it still does paging management internally just as kvm does. So direct memory management ( some HPC users like this ) becomes just as problematic as it is in kvm. Lots of overhead. I'm not convinced this is accurate. Can you provide some kind of reference for this? Okay so KVM uses a nastier abstraction layer in the form of shadow paging, while LXC simply relies on cgroups for memory isolation. Obviously two very different beasts. But there is the overhead of cgroup accounting and resource management inside LXC. Ah, yes. Cgroups. I'm obviously behind the times. In my head, an LXC container is just a namespaced (set of) process(es). Very good point. Thanks. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-compute] vm migration problem
2012/5/21 Lorin Hochstein lo...@nimbisservices.com: Has anybody ever written a script that grabs the host public key from the instance's console and updates the .ssh/config/known_hosts file accordingly, instead of throwing away host key checking? That would be a handy little thing if it was out there. Ubuntu's cloud-utils package has a cloud-run-instances utility that does this. It's not exactly in the do-one-thing-and-do-it-well sort of category, but perhaps it's just what you need. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] swift news and plans
2012/5/15 Andy Edmonds a...@edmonds.be: If I'm not mistaken: http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-05-08-20.00.log.html That meeting happened 4 days *after* the e-mail I responded to was sent to the mailing list. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [Swift] swift news and plans
2012/5/4 John Dickinson m...@not.mn: TL;DR: removing code from swift, associated projects doc, swift 1.5.0 This is interesting stuff. Where was this discussed? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Core Cleanup
2012/5/8 Monsyne Dragon mdra...@rackspace.com: Just wondering, who is maintaining the review day schedule? (http://wiki.openstack.org/Nova/ReviewDays) I can easily make time for more reviews if I tell folks it's my review day, but I have not seen my name on the schedule there for several weeks. I do. I have a script that takes care of it, but for various reasons (the fact that some people don't have any public e-mail adresses on Launchpad being high on the list) I have to maintain it manually. It would appear that you've somehow managed to escape the list. :) I'll rectify that the next time I generate the table. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Canonical AWSOME
23. apr. 2012 17.15 skrev Justin Santa Barbara jus...@fathomdb.com: What's the advantage of replacing the native EC2 compatibility layer with AWSOME from a user / operator point of view? Although I wasn't able to attend the design summit session, right now we have two native APIs, which means we have two paths into the system. Yes. This is good. It keeps the layers of separation honest. Multiple consumers of internal API's helps in making it more obvious at which layer functionality belongs. Just like having multiple hypervisor drivers (at least supposedly) makes it more obvious what sort of stuff belongs in the drivers and what sort of stuff belongs in the compute manager. That is poor software engineering, because we must code and debug everything twice. I beg to differ. If we need to fix things twice, it's because we've violated the layers of separation somewhere. For the record, I'm fully prepared to believe that we've done so. I also fully believe that the fact that we haven't done so *more* is because we have two API's to consider. Eric Day went through all the calls from the frontends into the various backend managers a long time ago, ensuring this separation was clear. I'm convinced that the result was a massive improvement. Some developers will naturally favor one API over the other, and so disparities happen. Today, both APIs are effectively using an undocumented private API, which is problematic. Yes. This is a problem. However, as I understand it, there was a session at the summit about versioning the internal API's. I'm not sure how we can usefully version the API's without also documenting them, so that problem should be adressed in the relatively near future. We also can't really extend the EC2 API, so it is holding us back as we extend OpenStack's capabilities past those of the legacy clouds. If EC2 API is limiting what we can do, that's not going to change just because you move the EC2 API implementation into a proxy in front of the OpenStack API. The only difference is that it's suddenly the AWSOME development team's problem. With one native API, we can focus all our energies on making sure that API works. Then, knowing that the native API works, we can build other APIs on top through simple translation layers, and they will work also. Other APIs can be built on top in the same way (e.g. OCCI) Sorry, I'm having trouble here.. Are you suggesting that having two sibling frontend API's talking to a shared backend API is poor software engineering, but layering similar purposed API's on top of each other like this is good software engineering? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] I18n issue for OpenStack
Den 12. apr. 2012 13.27 skrev Hua ZZ Zhang zhu...@cn.ibm.com: These messages always need to be localized, returned and displayed on user interface, not just be logged in backend system. It is very common practice for a global software project. OpenStack isn't an interactive application. It's a piece of server software. I'm struggling to think of any server software that writes anything in its log files in anything other than English. I'm Danish and generally configure my servers to speak Danish, so I should have seen it at least occasionally. Do you have any examples of this alleged very common practice? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] I18n issue for OpenStack
Don't get me wrong.. I'd be happy to have the various openstack clients offer localised error messages. I'd also encourage a centralised effort to collect these translationns (so that all the various language bindings will use the same localised error messages). On the server, though, I believe we should stick to English and perhaps have every error message include a link (e.g. http://docs.openstack.doc/exception/NoNetworksDefinedException) to a localised docs site. I think losing the ability to search the web for error messages would be a major loss. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] EC2 metadata
29. mar. 2012 00.55 skrev Joshua Harlow harlo...@yahoo-inc.com: I’ve seen that before. But was wondering if there was any secret other wiki. There is no secret wiki that I'm aware of. If you believe the referenced wiki page is out of date, please help update it. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Gerrit minimum review time frame
2012/3/12 Josh Kearney j...@jk0.org: Is this is really a problem that needs solving? I'd like to believe that no member of Nova Core would approve something that they aren't familiar with. That's not the point. The review process isn't just about finding enough people who agree with your change, it's just as much (some might argue, it's even more) about giving people a chance to *disagree*. IMHO, we should be making better attempts at not letting branches sit around for days/weeks at a time. I agree. That's a completely separate issue, though. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] speed up unittest in Nova
2012/3/9 Hengqing Hu huda...@hotmail.com: Last few weeks I spend some time to speed up unittest in Nova. This is great stuff! Thanks! -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] cobbler
2012/3/12 khabou imen imenkh...@gmail.com: I want to use cobbler between 2 server from ovh both of them has ubuntu server 11.10 as os in the first I installed orchestra and when I use cobbler to boot in PXE mode to the next machine i failed to use juju (the commande juju status didn't work) My question is :Is'it possible to boot in PXE mode on a machine which has ubuntu-sever 11.10 as OS or I must have a VM to be able to use cobbler Whether or not you can PXE boot is not dependent on the OS. It depends on the hardware (and firmware). -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Can openstack support spice ?
2012/3/11 wangsuyi640 wangsuyi...@gmail.com: Dear all: The release of openstack on my server is D3. I tred to run ' qemu-kvm -hda /root/free.img -m 512 -vga qxl -spice port=5930,disable-ticketing ' without openstack , it worked well. However,I have tried to modify the '/opt/stack/nova/nova/virt/libvirt.xml.template' in order to let the spice work with the openstack. Then it failed. Is there anyone tried this? Could you give me some help? Thanks. It's a bit hard to work out what is failing when you don't tell us what you actually changed. What does your changed libvirt xml template look like? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Running for Nova PTL
2012/2/28 Vishvananda Ishaya vishvana...@gmail.com: There is always progress to be made, but while we are toiling away working on testing, technical debt, and code smell, we have to keep our users in mind. This part of your e-mail really sticks out, along with this one: I disagree with this point. There was a huge amount of effort from a lot of parties on getting things cleaned up during essex. Most of the features that were worked on were getting things aligned and consistent behind our api. We have to make sure that Nova is usable as well as stable. It is tempting to think of things from the developer/code perspective, but we have to put on our user/operator hats as well. I'm trying hard to work out what you're implying here.. Are you suggesting that my suggested policy of not adding features, but focusing almost exclusively on stability isn't for the best of our users? Who do you think (I think) I'm favouring when I propose something like that? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Basic networking/configuration woes
2012/2/24 Justin Santa Barbara jus...@fathomdb.com: I have contributed a patch (which has merged) which should allow you to stop editing the SQL: https://review.openstack.org/#change,3816 With that, you should be able to pass the full range, with an additional argument specifying the subset that nova controls: e.g.-fixed_cidr=10.200.0.0/16 When I boot my VM, I think it gets a real address from my DHCP server (because the VM can reach the DHCP server), but not the address nova assigned it! I believe the nova iptables rules mean that the machine can't then do TCP/IP, but even if I am wrong/could overcome that, I don't think cloud-init could then configure the correct address. The instance firewall should be configured to only allow DHCP responses from the IP it believes to be the correct DHCP server. Perhaps it has the wrong idea? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Running for Nova PTL
2012/2/24 ed_con...@dell.com: I like most of what you say, but the no new features at all policy for trunk for the Folsom causes concern. I absolutely agree that Nova needs to be more stable, predictable and have more work done around operations. This is especially true for service provider usage. I think if Nova was closer to feature complete, that the no new features at all policy for trunk for the Folsom would be fine. It is probably doable for Folsom if the focus is on private cloud functionality. But for service providers, Nova is not feature complete and there is still much needed from the operations standpoint. I understand it's a trade-off. People may feel there are features missing before they can really use Nova, but noone can use it if it's unstable or doesn't work. I believe everyone will be better off if we spend some time (like, say, 6 months time) and focus exclusively on addressing these things. Thinking we can both add lots of new features as well as rework things for more stability at the same time is how we've ended up here. If you know of a way to achieve both, I'd love to hear about it. As much as I wish everyone would drop what they are doing and start helping in this effort, I understand some groups have certain features they simply cannot defer. I'd like to encourage this to happen collaboratively in public repos, and with frequent rebases against trunk to ease inclusion into trunk later on. In a way, they'd be patch sets more than anything else. How does that sound to you? -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Running for Nova PTL
2012/2/24 ed_con...@dell.com: That can work and may be the only choice if there is an extended feature freeze. Although, that may end up creating a service provider-specific fork...which may not be a bad thing. That's certainly not my intent nor desire. At least not permanently. I'd love to have a chat about the details of how we can make this work both short and long term. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Running for Nova PTL
2012/2/24 Eric Windisch e...@cloudscaling.com: On Friday, February 24, 2012 at 12:01 PM, ed_con...@dell.com wrote: That can work and may be the only choice if there is an extended feature freeze. Although, that may end up creating a service provider-specific fork...which may not be a bad thing. It can also be a very, very bad thing. Segmentation of the community and an exponentially increased complexity for those of us playing both sides of the private/public fence. I really can't see any advantage of forking, even temporarily. Forking has a number of connotations, not all of which necessarily strictly apply here. We have some fundamental problems we need to address. To do so effectively while still letting others move forward towards their goals, the proposal (in its current form) is to let people work on the things they need to meet their goals in a separate branch (probably in a separate repository), yet still under the Nova umbrella. Once we feel we've addressed the aforementioned problems, we can start merging the features in from these other branches. As I said, I'm expecting these experimental (or whatever designation is appropriate) branches will frequently be rebased on top of trunk so as to ease a later merge of the branches. The fact that these forks are still in many ways part of the project (albeit do not follow the rigour of the OpenStack release cycle), as well as the fact that they are expected to rebase often, make them fail to qualify as forks, at least in the traditional sense. I'm opposed to a broad Folsom feature freeze as it would too greatly limit progression. However, I also agree that there needs to be a better core focus, rather than on sprawl. I don't believe what Nova needs is features, be they core or in the various drivers. We need stability and dependability. I'm not opposed to selective feature inclusion. On the same token, I believe we should either approach the Linux kernel model of include the kitchen sink or not, and by not, Nova would be the core framework upon which various drivers would be provided and could be plumbed. Would you mind proposing this as a session for the summit? It sounds like a good chat to have face to face. One of the things I'd like for us to address in Folsom is actually to make this easier. Many parts of the nova code base can't easily be split out, but drivers should be fairly easy to maintain separately. We should make that simpler. This, like other things I've suggested, is mostly a job of clarifying contracts between various components. If today, for instance, it was announced that Folsom won't include new features, then it would be impossible for Coraid, Pillar, or some other storage solution provider to offer a driver in Folsom and would have wait until G. Yet, Nexenta just got their driver into Essex. Nexenta's 6 month lead just turned into a 12 month lead! Sure, their competitor could ship separately, but there *is* a difference between inclusion, and now, if only politically and from the perspective of marketing. That is a very good point, and like all good policies, exceptions can be made. A change that simply adds a new driver (and doesn't really touch any core parts of Nova at all) would be a very good candidate for an exception, particularly in the volume subsystem, which, last I looked at it, seemed quite well behaved. If new drivers and new code won't be accepted easily, then these third party drivers should be maintained as external plugins. While nice in theory, I don't agree with it at this time. These are contributions to OpenStack and are core, essential to the success of the community. Breaking out drivers, while easier, would fracture the community in potentially devastating ways. That's a valid point of view. However, when we need to weigh it against the burden of maintaining potentially unmaintainable plugins, it's not always an easy decision. We'd certainly need to clarify the social contracts better than we have in the past. So, in case we have to accept into core a driver for some piece of infrastructure we cannot easily get access to (and as a team don't really have a vested interest in), someone needs to be all over it. We need to make sure those expectations are clearly communicated. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Security Group Rule Refresh
2012/2/22 McNally, Dave (HP Cloud Services) dave.mcna...@hp.com: Currently I’m trying to track how a refresh of the security groups is handled (upon creation or deletion of a vm). Following through the code I get to ‘do_refresh_security_group_rules’ in libvirt/firewall.py. Up to this point the security group in question has been carried through however it seems to be discarded here and rather than filtering the instances to refresh the rules for based on this group it looks to me like all instances on the current host are iterated through and then there is an attempt to update the rules for all these instances. Is this full refresh necessary/intentional? If so can anyone tell me why it’s required? I forget the exact history here (i.e. why some of the method calls include it and why some don't), but there are three reasons I decided to do a full refresh: 1 deal with the situation where a refresh call to one of the compute nodes got lost. If that happened, at least it would all get sorted out on the next refresh. 2 the routine that turned the rules from the database into iptables rules was complex enough as it was. Making it remove only rules for a single security group or a single instance or whatever would make it even worse. 3 The difference in terms of efficiency is miniscule. iptables replaces full tables at a time anyway, and while the relative amount of data needed to be fetched from the database might be much larger than with a more selective refresh, the absolute amount of data is still pretty small. Point 1 could be addressed now by a periodical refresh of the rules, if one was so inclined. Point 2 should be more palatable now that the simpler implementation has proven itself. Point 3 might be less true now. In the beginning, there were separate chains for each security group, now it's just one big list, IIRC. That may change things. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Running for Nova PTL
I've put my name on the ballot for Nova PTL, and I'd like to explain what I expect to do (my platform, if you will). Nova is facing many separate, but related problems. * Nova is too big. Very few (if any) core developers are comfortable reviewing every part of the code base. In itself, this isn't necessarily a problem, but I think it would be valuable to try to somehow acknowledge that the average focus is much narrower than all of nova. * Lots of things in Nova that should be orthogonal are not. This problem is especially prevalent in the virtualisation layer. The layout and number of disks you get attached to instances shouldn't depend on the hypervisor you've chosen, but it does. There is lots and lots and lots of logic embedded in both the libvirt and XenServer drivers that isn't related to the hypervisor, but is a result of the origin of these drivers. * The overall quality is decreasing There's an almost unilateral focus on features across the board. The topic of almost every session at the summit is some new feature. There is very little focus on stability, predictability and operation. Personally, I think that shows very clearly in the final product. I'd like to try to shift our focus and turn the proverbial ship around. I'd like to remove any incentive to rush things into Nova trunk. 1. A much shorter release cycle (as Thierry also suggests[1]) would be very beneficial. Noone wants to have to wait an extra 6 months getting some new feature in just because it missed the feature freeze. However, just a single month of delay... That should be manageable in most cases. 2. I'd like to make it more straight forward to have things mature somewhere separete from Nova trunk, but still make it easy to collaborate on them or get people to test them. 3. I'd like to encourage a stronger focus on QA and testing. Specifically, I'd love to have more people focused on making it easier to test things in Nova. Tempest is a great effort, but the unit test suite is our first line of defence. It should be fast and comprehensive. Right now, it's neither. 4. I'd like a stronger focus on extensibility and plugability. 5. I'd like us to rethink our configuration management strategy. So far, we've punted on it and deferred to deployers to choose between Puppet, Chef or whatever else to handle this. However, many things will crash and burn if the configuration of various components is out of sync with each other or with the database. This is particularly clear in the networking area. [1]: http://fnords.wordpress.com/2012/02/21/open-dev-releases-quality/ -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Running for Nova PTL
2012/2/23 Duncan McGreggor dun...@dreamhost.com: Soren, if elected, by what processes/policies etc. would you accomplish these goals? Well, there are limits to what a PTL really can do :) However, in practical terms there are a number of things I'd like us to do: * I'd like us to look at the various components of Nova and thoroughly document (in prose as well as as tests) their API and expected behaviour. It's very tempting to change (in major or minor ways) these API's on a whim since we control both ends of the channel (often even in the same patch), but this a distributed system. Upgrades across an entire Nova installation are not instantaneous, and shouldn't have to be. We need to be more aware of the interfaces between components and the fact that they may not be in perfect sync. * In a similar vein, while we do a good job ensuring db schema upgrades work well, the code doesn't support anything other than the newest schema it knows about. Or rather, if it does, it's by accident. This makes it exceedingly difficult to upgrade a Nova installation peacemeal. * I'd like to revamp the virtualisation subsystem to move much more behavioural logic into a superclass and have the actual drivers be only the glue code to make the individual hypervisors work. * As I wrote in my response to Robert earlier in this thread, I'd like to see more branches pop up specific to particular subsystems. I'd like to make it easier to get features landed somewhere and let them mature there before they hit trunk. * I'd like to have much more frequent releases, and I mean *actual* releases, not just milestones. Each with merge windows, QA phases, release artifacts, etc. * Lots of other things I'll try to elaborate on over the next few days. Are there blueprints that already exist which you would rally folks around? Or would you introduce a new effort to more thoroughly componentize OpenStack? More specifically, how do you envision: 1) clarifying what needs to be done I don't expect to do this all on my own. I'd like to set some overall topics for the release cycle and try to seed the conversations about these efforts (as I'm trying to do right now), but I'd really, really like for everyone else to help identify all of this stuff and find issues you care about. 2) building consensus around this, and Excellent question. I can't force anyone to suddenly think QA and unit tests are the most important things in the world. :) I think there's a strong correlation between my chances of getting elected and the how much of a pre-existing consensus there is around the issues (and issues like them). If I get elected, it's because people already think these things are important, so it shoulnd't be too hard. Or so I hope. 3) accomplishing these goals? (it's a lot of work!) I hope the rest of my e-mail sheds a bit of light on this. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Security Group Rule Refresh
2012/2/23 Day, Phil philip@hp.com: 1 deal with the situation where a refresh call to one of the compute nodes got lost. If that happened, at least it would all get sorted out on the next refresh. Can see the advantage of this, but on an active system this can be quite an overhead compared to a periodic refresh. Well, a periodic refresh will likely happen more often than the refreshes triggered by changes, don't you think? And periodic refreshes will inevitably have to refresh everything (otherwise they seem pointless). 2 the routine that turned the rules from the database into iptables rules was complex enough as it was. Making it remove only rules for a single security group or a single instance or whatever would make it even worse. I wonder if we're talking about the same driver - the code we're looking at is in the IptablesFirewallDriver in libvirt/firewall.py (which I think is moved up to virt/firewall.py in Essex). That seems to create a chain per Instance and do the update on a per instance basis, so I'm not quite sure I understand your point ? Sorry, I was basing this all on memory. The point is simply that if the routine that did all of this would have to reliably leave everything else alone, and only touch the rules pertaining to a particular instance, the logic would be even more complicated than it already is. 3 The difference in terms of efficiency is miniscule. iptables replaces full tables at a time anyway, and while the relative amount of data needed to be fetched from the database might be much larger than with a more selective refresh, the absolute amount of data is still pretty small. It may be that we're hitting a particular case - but we have a test system with 10's of VMs per host, on not many hosts, and some groups with 70+ VMs and a rule set that references the security group itself. So every VM in that group that gets refreshed (and there are many on each host) has to rebuild rules for each VM in the group. That's a bug. It's supposed to only refresh once, regardless of how many affected VM's there are. The impact of this overhead on every VM create and delete in un-related groups is killing the system - eps as the update code doesn't yield so other tasks on the compute node (such as the create itself are blocked). Have you been able to profile this at all? Is it the DB query that takes a long time or is it something else? Anyways, I don't fully understand why any part of the process would make anything hang. Both the communication with the DB as well as calling out to iptables-restore should yield control over to the eventlet main loop and let other things run. I wonder why this isn't happening. Point 2 should be more palatable now that the simpler implementation has proven itself. Could you clarify which simpler implementation your referring to It's probably a poor choice of words :) The simpler implementation is the current one. The more complicated one would be one that reliably would only touch the rules pertaining to the instances or security groups that are actually being changed. - I've seen the NWFilterFirewall class and its associated comment block, but it wasn't clear to me under what circumstances it would be worth switching to this ? None, at the moment, due to this bug: https://bugzilla.redhat.com/show_bug.cgi?id=642171 -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Wish: Please rename all OpenStack packages to openstack-*
2012/2/22 Zhongyue Luo lzye...@gmail.com: Before we discuss whether the packages should be renamed or not, isn't it unorthodox to have a hyphen in a package name? Out of the 38480 packages available on amd64 in Ubuntu Precise, 28910 have a hyphen in their package name. -- Soren Hansen | http://linux2go.dk/ Senior Software Engineer | http://www.cisco.com/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] confused about libvirt nwfilter and iptables rules
The original implementation of this filtering used only nwfilter. Due to shortcomings in nwfilter in libvirt and netfilter in the Linux kernel, this turned out not to work very well at all, so an alternate implementation using raw iptables was added. This is now the default. However, nwfilter works excellently at protecting against MAC spoofing, ARP spoofing and IP spoofing, so we still use it for that. Does that help? 2012/2/7 heut2008 heut2...@gmail.com: hi,all: I am confued about how security rules works ,i read the /nova/virt/libvirt/firewall.py and /nova/network/linux_net.py , my understanding is when create or change a security rule ,the process is as below. reuqest to nova osapi-update db for the rule-call method trigger_security_group_rules_refresh()-rpc.cast to all reletave compute node. -call refresh_security_group_rules(),it seems that refresh_security_group_rules get the rule from the db and use libvirt to define the rules. but how iptables are invoked to create rules like nova-compute-inst-22. anther question is libvirt defines nova-base-filter which allow any packets out and drop all packets in ,but it does not used by the instance nwfilter. the instance nwfilter only has no-mac-spoofing ,no-arp-spoofing,no-ip-spoofing ,and allow-dhcp-server filter. if I misunderstand some thing ,please correct me ,thks . ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack Foundation] OpenStack Foundation
It is now Feb 4th. The has been no updates since Jan 5th. Not that I've noticed, at least. The schedule on http://wiki.openstack.org/Governance/Foundation says a revised (or final?) mission statement would have been posted last week. Also, a draft structure should have been posted this week? Can we still expect that? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack Foundation] OpenStack Foundation
2012/2/4 Soren Hansen so...@linux2go.dk: The schedule on http://wiki.openstack.org/Governance/Foundation says a revised (or final?) mission statement would have been posted last week. Ah, since I wrote my e-mail yesterday, it seems that page has been updated. Sorry. We were promised more communication. I think the absolute minimum would be announcing changes to the schedule on the mailing list. More info about what is actually going on would also be nice. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack Foundation] OpenStack Foundation
2012/2/4 Soren Hansen so...@linux2go.dk: 2012/2/4 Soren Hansen so...@linux2go.dk: The schedule on http://wiki.openstack.org/Governance/Foundation says a revised (or final?) mission statement would have been posted last week. Ah, since I wrote my e-mail yesterday, it seems that page has been updated. Sorry. I also see that the DRAFT FOR COMMENT designation at the top of the Mission page was removed a couple of days ago[1]. Is that intentional? [1]: http://goo.gl/e8lHf -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Libvirt File Injection
2012/1/30 Brian Waldon brian.wal...@rackspace.com: After implementing a working version of file injection on Libvirt, a good question was brought up on the merge prop: how should we handle a file injection failure? Injection could fail for several reasons: missing necessary libraries, unsupported image formats and bad permissions are just a few. There seem to be two clear paths forward: 1) Log an error, set the instance to ERROR, add an asynchronous fault to the instance in the db 2) Log a warning, move on with the boot process It's not obvious which of these is the best route to take from a user's point of view. I'm currently leaning towards option 1 as I wouldn't want to have an instance come up (and be billed for it) while it wasn't what I explicitly requested. I agree with this. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Libvirt File Injection
2012/1/30 Dan Prince princ...@alumni.jmu.edu: On Mon, Jan 30, 2012 at 1:05 PM, Brian Waldon brian.wal...@rackspace.com wrote: After implementing a working version of file injection on Libvirt, a good question was brought up on the merge prop: how should we handle a file injection failure? Injection could fail for several reasons: missing necessary libraries, unsupported image formats and bad permissions are just a few. There seem to be two clear paths forward: 1) Log an error, set the instance to ERROR, add an asynchronous fault to the instance in the db 2) Log a warning, move on with the boot process My preference would be to log a warning and move on with the boot process (#2). Or perhaps we could address this with some sort of async error messages concept? If getting those files injected isn't critical to getting the machine up and running, you can use one of the many other ways to get data into your instances. If the API calls says to inject a file, and we know that failed, we should bail out. I certainly wouldn't want to pay for something that isn't going to work. Sure, most of the times when I've booted things on EC2 or on the Rackspace cloud, I've done so interactively and have had the chance to see any errors, but that's going to be more and more of a niche use case. I fully expect the majority of consumer of OpenStack in the future to be applications wanting more ressources. If we can't deliver on their requests, we should fail early so that they can take appropriate action ASAP. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] What's the web server used in the Nova API server?
It's not like we wrote our own webserver :) Eventlet has a wsgi container. We just went ahead and used it. Not only does it perform very well, it's also very straightforward to use for testing (compared to having to install and configure Apache to get going).. 2012/1/16 Michael Basnight mbasn...@rackspace.com: Just curious, whats the reason we went with rolling our own instead of using something like nginx/apache2/etc w/ mod_wsgi? On Jan 16, 2012, at 2:14 AM, Thierry Carrez wrote: Joe Smithian wrote: I browsed the openStack documentation but couldn't find information about the Nova API server. What's the web server used in the Nova API server? Can we use a different web server such as Apache or Tomcat? I'd appreciate your comments. Nova uses Python eventlet WSGI servers. You can't directly use a different web server, though you can certainly place Nova API servers behind some other server. -- Thierry Carrez (ttx) Release Manager, OpenStack ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [OpenStack Foundation] OpenStack Foundation
2012/1/5 Jim Curry jim.cu...@rackspace.com: So now for the update. In reality, there has been a ton of work occurring to build a proposal that can serve as the basis for a constructive community conversation. I firmly believe that the best way to have a conversation with a group as diverse and large as the OpenStack community is to start with a very defined proposal that everyone can react to — positively or negatively. This proposal is not meant to be anything other than a framework to manage the discussion. No decisions are being made without the involvement of the broad community. What does that mean? From what I understand so far, someone at Rackspace will eventually publish some documents for discussion and will listen to feedback. What if this feedback isn't unanimous? Are shooting for a minimum that we can reach consensus on (by ripping out everything that anyone at all disagrees with) or will there be a vote on individual points of contention? Will the documents as a whole be put to a vote? If so, what if they're rejected? In both cases, who gets a vote? Trying to handle the large number of complex issues over a mailing list without a base framework would be impossible to manage. It is also unfair to community members who don't live on the mailing lists day to day, but do want to review the plans and provide comments. I'm much more concerned about the quality of the feedback than the quantity. Twitter only lets you make a point in 140 characters. That's hardly enough for a delicate discussion like this. A webinar isn't any better, because these things need carefully thought out arguments, not just whatever you can come up right there on the spot. Honestly, if people have strong opinions on this subject, but can't be bothered to send an e-mail, that's sad. We are basing this initial proposal on a lot of input received from the community and beyond — developers, users, companies, other open source projects and foundations, lawyers, specific country experts, etc. How has this worked? Have these people (who?) been asked a specific set of questions? Can we see them so that we can consider the same questions? Can we see the feedback itself? If you expect the foundation after it's been established to take it into consideration at all, it needs to be disclosed, otherwise the first revision of the bylaws/constitution might go against this feedback entirely, and then it's been a waste of time. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Metadata and File Injection
2012/1/2 Ewan Mellor ewan.mel...@eu.citrix.com: In the context of this discussion, that means that OpenStack needs to work in DHCP-free environments, because we already know of many. The only one I'm familiar with is Rackspace where I think (and please do correct me if I'm wrong) DHCP isn't used only because Windows instances can't be configured with DHCP if they have multiple interfaces. DHCP may be forbidden because: * it has been deemed less reliable than pushing fixed IP addresses from an asset database. If you already have a comprehensive asset database, why push the addresses to a DHCP server to have it push them to your hosts when you can push them to the host directly? Because Nova isn't likely to interface with your asset database. It maintains has its own database for these things. Also, how do you push something from an asset database, if you don't have network connectivity? * it has been deemed less reliable than pushing addresses from the virt layer. By whom and for what reasons? Every single operating system of any relevance at all understands DHCP. It's an extremely widely used system. It may not be perfect, but that will hold true for any replacement we decide to build. You'll simply be replacing the bugs, problems and idiosyncracies with a different set of bugs, problems and idiosyncracies. Why go to the effort of making a DHCP service highly available, and making sure that the failover is quicker than the timeout that causes guests to decide to take zeroconf addresses instead? Just set the lease time to a couple of weeks. If you can't manage to fix a dead dnsmasq within that timeframe, that's probably the least of your problems. * it has been deemed a security risk. Compromising the DHCP server gives you all sorts of easy attacks. The DHCP server runs on the network controller. If your network controller is compromised, you're pretty screwed anyway, afaics. This one actually seems like a very reasonable concern in the case of tenant networks in a cloud. Pushing the address from the virt layer takes away that attack vector altogether. ...and replaces it with another. * the guy who is installing the cloud doesn't control the network, and so the DHCP server isn't his and/or he's not allowed to run one of his own. This seems likely in an academic environment or pilots within an enterprise. a) This isn't a reason. It's just repeating the original non-problem: it's not allowed. b) There are several ways in which we can ensure that our DHCP stuff doesn't bleed into the rest of the network. dnsmasq already won't respond to requests from mac addresses that it doesn't know about, but if that isn't good enough (why would this be? what's so darned special about dhcp requests anyway?), we can add extra strict filtering in a couple of places. Note that none of these has to be specifically a decision about OpenStack or clouds in general. They might just be blanket policies from simpler times. That doesn't make them any easier to change. I can't tell you how to run your business. All I know is that if a client of mine gave me a functional requirement specification that would be perfectly met by DHCP, but they had a piece of paper from the mid-90's that said DHCP isn't allowed. Just because. on which they refused to budge, I'd pass on the contract. I personally don't want to facilitate stupid policies. That's my policy. Regardless of those four, your one seems like a complete deal-breaker to me. If that's true (and it's not something that I'm aware of one way or another) then we shouldn't even be having this discussion. We can't build a solution that can't reliably configure Windows VMs. a) I don't think we should generally limit ourselves to whatever Windows can support. (Much like how I don't think we should have a hard dependency on hypervisors that have a Xenstore-like backdoor into the guest) b) I don't think I've heard any reason why we couldn't just configure one of the NIC's over DHCP, and then fetch the remaining network config from a network-connected metadata service. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Problems with run_tests.sh on 11.10
2012/1/2 Monty Taylor mord...@inaugust.com: Do you think someone would be willing to accept that patch? Or should we make our own branch with the patch applied and reference that in the pip-requires? If this is something we want to use for anything half serious, I think we should create a branch of our own (or pester upstream hard enough that they finally get around to fixing it). -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Glance functional tests failing
2012/1/1 Ewan Mellor ewan.mel...@eu.citrix.com: How has a keystone change managed to break Glance when we're pinning Keystone at a specific version? Could you try blowing away your .venv directory and trying again? Perhaps there's a stale Keystone lying around in there. It won't get replaced, because it already fulfills the stated requirement (given by egg=keystone). Perhaps we should change pip-requires to read egg=keystone-dev or something. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Configure Rate limits on OS API
2011/12/29 Vishvananda Ishaya vishvana...@gmail.com: In addition to the rate limiting changes, you can also run multiple copies of nova-api on different ports and put a load balancer (like ha-proxy) in front. You may have to set up your proxy to add an X-Forwarded-For header and specify the --use_forwarded_for flag to get things like metadata to work. I can't think of any reason we can't just fork() N times (where N defaults to number of cores, but could be overridden by a flag) after socket.listen(), but before we go into the socket.accept() loop? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Problems with run_tests.sh on 11.10
2011/12/30 John Griffith john.griff...@solidfire.com: Oops, sorry about that. Forgot to check it in the venv, which reveals the issue: % tools/with_venv.sh jdg@grumpy ~/Projects/OpenStack/nova % python Python 2.7.2+ (default, Oct 4 2011, 20:06:09) [GCC 4.6.1] on linux2 Type help, copyright, credits or license for more information. import M2Crypto Traceback (most recent call last): File stdin, line 1, in module File /usr/local/lib/python2.7/dist-packages/M2Crypto/__init__.py, line 22, in module import __m2crypto ImportError: /usr/local/lib/python2.7/dist-packages/M2Crypto/__m2crypto.so: undefined symbol: SSLv2_method Ah, yes. That's because M2Crypto hasn't kept up wit the removal of SSLv2 from OpenSSL. It's fixed in the Ubuntu packages, so if you remove the M2Crypto line from pip-requires and put this instead: -e bzr+http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/precise/m2crypto/precise/#egg=M2Crypto You should be fine. (Yes, the line in pip-requires should start with -e) -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Integration test gating on trunk
2011/12/29 James E. Blair cor...@inaugust.com: Having said that, the Jenkins job has been running in silent mode on master for several days with few false errors. My feeling from the design summit was that it was generally understood there would be a shakedown period, and people are willing to accept some risk and some extra work for the benefits an integration test gating job will brink. I think we're at that point, so I'd like to turn this job on Tuesday, January 3rd. I'm much happier with a test rig that has the occasional false negative than no testing at all. A big +1 from me. Thanks for this excellent effort! -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Problems with run_tests.sh on 11.10
2012/1/2 Lorin Hochstein lo...@isi.edu: Would this work for Fedora-based distributions as well? Yes. The source checkout will include a debian/ directory, but that won't affect the Python code installed. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Quantum OVS over StackOPS
2011/12/21 Alisson Soares Limeira Pontes apon...@cpqd.com.br: I am configuring nova to work with Quantum. When aplying the configurations for the OVS plugin I face this error. root@nova-compute-1:~# python ovs_quantum_agent.py ovs_quantum_plugin.ini Traceback (most recent call last): File ovs_quantum_agent.py, line 297, in module plugin = OVSQuantumAgent(integ_br) File ovs_quantum_agent.py, line 175, in __init__ self.setup_integration_br(integ_br) File ovs_quantum_agent.py, line 188, in setup_integration_br self.int_br.remove_all_flows() File ovs_quantum_agent.py, line 89, in remove_all_flows self.run_ofctl(del-flows, []) File ovs_quantum_agent.py, line 86, in run_ofctl return self.run_cmd(full_args) File ovs_quantum_agent.py, line 58, in run_cmd p = Popen(args, stdout=PIPE) File /usr/lib/python2.6/subprocess.py, line 633, in __init__ errread, errwrite) File /usr/lib/python2.6/subprocess.py, line 1139, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory Anyone know why? The file /usr/lib/python2.6/subprocess.py does exists! All steps before this (according to the OVS README) went well. I am using Quantum 2012.1-e2-ubuntu2 and the StackOPS0.3. It's complaining that it can't find ovs-ofctl. Did you install openvswitch? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Quantum OVS over StackOPS
Please share the exact error message from after installing openvswitch. Best regards, Soren Hansen 2011/12/21 Alisson Soares Limeira Pontes apon...@cpqd.com.br When i first had the problem i did not have it installed. But then i installed it root@nova-controller:~# ovs-ofctl -V ovs-ofctl (Open vSwitch) 1.3.0 Compiled Dec 20 2011 18:08:01 OpenFlow versions 0x1:0x1 but the problem persists. 2011/12/21 Soren Hansen so...@linux2go.dk 2011/12/21 Alisson Soares Limeira Pontes apon...@cpqd.com.br: I am configuring nova to work with Quantum. When aplying the configurations for the OVS plugin I face this error. root@nova-compute-1:~# python ovs_quantum_agent.py ovs_quantum_plugin.ini Traceback (most recent call last): File ovs_quantum_agent.py, line 297, in module plugin = OVSQuantumAgent(integ_br) File ovs_quantum_agent.py, line 175, in __init__ self.setup_integration_br(integ_br) File ovs_quantum_agent.py, line 188, in setup_integration_br self.int_br.remove_all_flows() File ovs_quantum_agent.py, line 89, in remove_all_flows self.run_ofctl(del-flows, []) File ovs_quantum_agent.py, line 86, in run_ofctl return self.run_cmd(full_args) File ovs_quantum_agent.py, line 58, in run_cmd p = Popen(args, stdout=PIPE) File /usr/lib/python2.6/subprocess.py, line 633, in __init__ errread, errwrite) File /usr/lib/python2.6/subprocess.py, line 1139, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory Anyone know why? The file /usr/lib/python2.6/subprocess.py does exists! All steps before this (according to the OVS README) went well. I am using Quantum 2012.1-e2-ubuntu2 and the StackOPS0.3. It's complaining that it can't find ovs-ofctl. Did you install openvswitch? -- Soren Hansen| http://linux2go.dk/ Ubuntu Developer| http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ -- Alisson Pontes __ Network Technology Evolution Researcher CPqD - Center for Research and Development in Telecommunications Tel.: +55 19 3705-4996 apon...@cpqd.com.br apo...@cpqd.com.br www.cpqd.com.br -- Soren Hansen| http://linux2go.dk/ Ubuntu Developer| http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] using objects returned from DB layer
2011/12/15 Jesse Andrews anotherje...@gmail.com: I agree except I though the preference was for instance_uuid = instance['uuid'] not instance_uuid = instance.uuid (use dict's and don't assume sqlalchemy) Yes... That's what Chris is saying, isn't it? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Subteam Changes
2011/12/11 Stefano Maffulli stef...@openstack.org: Any time you find will be in someone's evening and in someone else's middle-of-the-night, and most of us probably also have busy calendars. What is not clear to me is if you would like to reduce the amount of meetings or just not add more. One step at a time :) Which ones of the meetings listed on http://wiki.openstack.org/Meetings/ are the most difficult for you to attend to? I'm sorry that I can't attend the QA team meeting, the orchestration team meeting and the nova-db team meeting. I've accepted that my Tuesday evening is sacrificed on the altar of IRC meetings. After all, it's only the evening and only one of them. I feel quite sad that it's virtually impossible for anyone East of here to attend as it'll be during their night, though. My business hours are 9 - 17 (5 PM) CET. In my experience many employers are happy to give brilliant and experienced developers like you ample flexibility in terms of business hours, offering compensation (monetary, free time, and other perks). Are these hours your choice or are they mandated by your employer? I define my own working hours, but that's really beside the point. Calling from 2 AM - 3 AM working hours doesn't make it a convenient time slot, even if that means I take an hour off at a different time. No matter what you call it and how much time you take off at another time, it still means you have to get up in the middle of the night for a meeting. And if I have a meeting mid-evening, that evening is lost. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Subteam Changes
2011/12/10 Trey Morris trey.mor...@rackspace.com: This problem has to have been solved before. Anyone know how multinational companies deal with meetings? The distro team meetings at Canonical used to rotate 8 hours every week to distribute the pain. Eventually, though, we gave up having these big meetings altogether. Also, Soren, to be fair, I'm don't believe I've heard anyone propose a meeting that wasn't on ~utc-7 business time, yourself included. No, because I try not to propose meetings at all :) Maybe there should be some give and take here? Perhaps this is where give starts.. What time works for you? It certainly does seem fitting that the most pain falls upon the person proposing the meeting. Let's call it an incentive for people to not call meetings :) My business hours are 9 - 17 (5 PM) CET. In Central Time, that's 2 AM - 10 AM. In Pacific Time, that's 12 AM - 8 AM. In Japanese time, that's 17 - 1 AM. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Subteam Changes
On Dec 10, 2011 1:13 AM, Paul Voccio paul.voc...@rackspace.com wrote: I'm not sure why this frustration is directed at me. Because you're talking about calling a meeting as being no big deal, which might be true if everyone were in the same office, but when we're spread across the planet, it *is* a big deal. My point is sometimes you need to get people to make a decision and waiting for a week for everyone to toss their .02 cents doesn't always work. Now I'm confused. I thought you were arguing in favour of weekly meetings while I was arguing in favour of just raising subjects on the mailing list whenever you wanted. What are you arguing in favour of? If the idea is to have subteams who care about small parts of code then I should be able to go find those people to make a decision. Sure. And you can. We're right here on this very mailing list. In a project that is spread this far across the planet, you simply cannot expect to be able to just call a meeting and have everyone turn up. Any time you find will be in someone's evening and in someone else's middle-of-the-night, and most of us probably also have busy calendars. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Subteam Changes
2011/12/8 Vishvananda Ishaya vishvana...@gmail.com: I'm hesitant to do more meetings as well, but we do need some coordination. I call false dichotomy on this. I don't buy the without meetings, we can't coordinate anything notion. I will leave the one for Monday on the board for now. Lets have one meeting with the goal being to never need another one. Ok. Let me know how it goes. :) As I said, I can't attend. IDEA: We could have a status etherpad that everyone could keep updated each week with current tasks, important blueprints, concerns etc. What would that gain us compared to a status e-mail? The benefits of using e-mail instead of Etherpad: * They're easier to look up after the fact (mailing list arcvhies are already set up). * It enables dialogue about the contents. * Having to click the send button motivates you to actually finish it. Compared to IRC meetings: * No need for everyone to be in the same place at the same time. * No need for everyone to sit through a meeting where only a fraction of the topics are relevant to them. * Mailing list archives makes it simple to look up past discussions. * Much looser time limits. If I need to test a hypothesis before I make a statement, I can do that on my own time. That's very hard to do during a synchronous meeting. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Subteam Changes
2011/12/9 Paul Voccio paul.voc...@rackspace.com: I think the benefits of an all hands irc/irl meeting is to reduce the overall amount of time needed to drive to a decision. I can usually do this in a 30 minute meeting if I have the relevant people and have a handful of items that need to be decided upon. Having to have that same conversation with people usually takes a few days as email is not usually treated as a real-time medium. Ok, let's assume that a decision will always be reached if a topic is on the agenda for a meeting. Let's also assume that points for discussion can pop up at any given time throughout the week. (For the record, I only believe that one of these assumptions is sound, but meh.) As such, with a weekly meeting, it can take anywhere between 0 and 168 hours (or an average of 84 hours) to reach a decision. I honestly don't see how that's better than a predictable few days. This project is bigger than your office. You simply can't expect to have all the relevant people. Ever. How much Japanese attendance do we have at our meetings? It's exceedingly frustrating that this part of my objection is completely ignored: It will cost me an evening and will be in the middle of the night for Japan. You guys can spend each other's *working hours* on meetings all day long for all I care. This is (supposed to be) my spare time. We put *every* single meeting in this project in US business hours, *every* single meeting *outside* European and Japanese business hours, and *every* single design summit in the US and we're surprised there's a strong US bias? Seriously? Also, I stand by my other arguments against synchronous meetings and for e-mail discussions: Compared to IRC meetings: * No need for everyone to be in the same place at the same time. * No need for everyone to sit through a meeting where only a fraction of the topics are relevant to them. * Mailing list archives makes it simple to look up past discussions. * Much looser time limits. If I need to test a hypothesis before I make a statement, I can do that on my own time. That's very hard to do during a synchronous meeting. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Subteam Changes
2011/12/7 Vishvananda Ishaya vishvana...@gmail.com: 1) Weekly meeting for team leads. This is a time for us to discuss blueprint progress, multiple-team-related issues, etc. Going to shoot for Mondays at 2100 for this one. I really need the subteam leads to commit to making this meeting. We can discuss at the first meeting and decide if there is a better time for this to occur. I have a conflict in that time slot every other week, and frankly, I'd really like to avoid more meetings. I'd by far prefer keeping to e-mail, and if we (over e-mail) find something that warrants more real-time discussion, we can put it on the agenda for our regular openstack team meeting on Tuesdays. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova Subteam Changes
2011/12/8 Thierry Carrez thie...@openstack.org: Soren Hansen wrote: 2011/12/7 Vishvananda Ishaya vishvana...@gmail.com: 1) Weekly meeting for team leads. This is a time for us to discuss blueprint progress, multiple-team-related issues, etc. Going to shoot for Mondays at 2100 for this one. I really need the subteam leads to commit to making this meeting. We can discuss at the first meeting and decide if there is a better time for this to occur. I have a conflict in that time slot every other week, and frankly, I'd really like to avoid more meetings. I'd by far prefer keeping to e-mail, and if we (over e-mail) find something that warrants more real-time discussion, we can put it on the agenda for our regular openstack team meeting on Tuesdays. The problem is the general meeting has less and less time to dedicate to each project, so Vish identified that he could not get efficient subteam feedback and inter-subteam coordination from it... hence the need for a new meeting slot. It could be a short one (30min) at start. I hear that, but I'd like us to at least *try* and get it to work without stealing yet another one of my precious evenings for another horribly inefficient meeting. There are lots and lots and lots of thing we can do to avoid meetings that we haven't even tried. As a start, team leads could send a weekly report about what the team has been up to, what the challenges have been, and what they'll be doing the next week. When I say horribly inefficient meeting I'm not implying that our meetings are more inefficient than anyone else's, but just that meetings in general are horribly inefficient. The amount of information per second is tremendously low, and everyone has to sit through the whole thing even if they only care about a tiny fraction of the agenda (if they're even so lucky to have a proper agenda). Can we try to identify what sort of information we want from the various teams and *then* figure out how to extract and distribute it rather than just throwing a meeting slot at it? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [GLANCE] Proposal: Combine the container_format and disk_format fields in 2.0 Images API
2011/12/6 Jay Pipes jaypi...@gmail.com: On Fri, Dec 2, 2011 at 3:48 AM, Soren Hansen so...@linux2go.dk wrote: 2011/12/1 Jay Pipes jaypi...@gmail.com: There are basically two things that are relevant: The image type and the container format. The image type can be either of kernel, ramdisk, filesystem, iso9660, disk, or other. What value does other give the caller? It's meant to denote something that isn't a kernel, ramdisk, filesystem, iso9660, nor disk. Maybe swap space, maybe a raw block device for Oracle.. Something that's distinct from the other things, but isn't common enough to warrant its own designation. The container type can be: raw, cow, qcow, qcow2, vhd, vmdk, vdi or qed (and probably others I've forgotten). What about OVA? As I understand it, OVA is the single, tar'd file format that may store one or more virtual appliances that are in formats like VHD. I consider that a transport format. Maybe my choice of nomenclature is off, but an OVA clearly (based on your description) holds a number of somethings which in turn holds (typically) disk images. I'd much rather if Glance would extract all the relevant information from the OVA, store the disk images (setting the appropriate type and format in the process) and then discard the OVA. Much like how we treat ami's. It's a transport format. How does the following sound? Would this work for folks? type field, with the following possible values: * kernel * filesystem * ramdisk * disk * iso9660 Sure, I can live without the other type. format field, with the following possible values: * raw - This is an unstructured disk image format [...] * qcow2 - A disk format supported by the QEMU emulator that can expand dynamically and supports Copy on Write (Already responded about OVA). You're missing qed, qemu's next gen disk format. Should there be another format value of: * iso - An archive format for the data contents of an optical disc (e.g. CDROM). to correspond to the iso9660 type? No, iso isn't a format in the same sense as vmdk and qcow2, etc. Or should images with the iso9660 type have a raw format value? Yes, your garden variety .iso is a raw formatted iso9660 filesystem. It could technically be converted to any of the other formats, but seeing as they're tightly packed (no need for sparseness) and read-only (no need for sparseness nor copy-on-write), there's rarely much gained from that (other then confusion). -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Proposal for Mark McLoughlin to join nova-core
2011/11/29 Vishvananda Ishaya vishvana...@gmail.com: Mark is maintaining openstack for Fedora and has made some excellent contributions to nova. He has also been very prolific with reviews lately. Lets add him to core and make his reviews count towards potential merges! It's now been 6 days, there's been more than 5 +1's and noone has objected. Welcome to the team, Mark! -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Proposal for Lorin Hochstein to join nova-core
2011/11/29 Vishvananda Ishaya vishvana...@gmail.com: Lorin has been a great contributor to Nova for a long time and has been participating heavily in reviews over the past couple of months. I think he would be a great addition to nova-core. It's now been 6 days, there has been more than 5 +1's and noone has objected. Welcome to the team, Lorin. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [GLANCE] Proposal: Combine the container_format and disk_format fields in 2.0 Images API
2011/12/2 Donal Lafferty donal.laffe...@citrix.com: I simply don't think adding a vendor part to the container type string is going to be a very good way to encode this. Can this even be done with a MIME-style categorization? Perhaps the finer details of what MIME-style categorization is are lost on me. Can you elaborate? Your original example was vhd/x-ms-tools which, to my eye, is simply a container type string with a vendor part added. What am I missing? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [GLANCE] Proposal: Combine the container_format and disk_format fields in 2.0 Images API
2011/12/5 Donal Lafferty donal.laffe...@citrix.com: Perhaps the finer details of what MIME-style categorization is are lost on me. Can you elaborate? Your original example was vhd/x-ms-tools which, to my eye, is simply a container type string with a vendor part added. What am I missing? 'vhd' isn't a container type. It's a disk format. See http://glance.openstack.org/formats.html Err.. a) What on earth counts as a container type then? b) That's really beside the point. It's a something with a vendor part added. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] HPC with Openstack?
2011/12/4 Lorin Hochstein lo...@isi.edu: Some of the LXC-related issues we've run into: - The CPU affinity issue on LXC you mention. Running LXC with OpenStack, you don't get proper space sharing out of the box, each instance actually sees all of the available CPUs. It's possible to restrict this, but that functionality doesn't seem to be exposed through libvirt, so it would have to be implemented in nova. - LXC doesn't currently support volume attachment through libvirt. We were able to implement a workaround by invoking lxc-attach inside of OpenStack instead (e.g., see https://github.com/usc-isi/nova/blob/hpc-testing/nova/virt/libvirt/connection.py#L482. But to be able to use lxc-attach, we had to upgrade the Linux kernel in RHEL6.1 from 2.6.32 to 2.6.38. This kernel isn't supported by SGI, which means that we aren't able to load the SGI numa-related kernel modules. Why not address these couple of issues in libvirt itself? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [GLANCE] Proposal: Combine the container_format and disk_format fields in 2.0 Images API
2011/12/1 Jay Pipes jaypi...@gmail.com: structure tar'd up. However, I think this can be more easily accomplished by consolidating the disk and container formats in the 2.0 API to just a single format field with the possible values: ova - This indicates the data stored in Glance is an OVF container that may actually contain multiple virtual appliances that has been tar'd into the single-file OVA format raw - This is an unstructured disk image format vhd - This is the VHD disk format, a common disk format used by virtual machine monitors from VMWare, Xen, Microsoft, VirtualBox, and others vmdk - Another common disk format supported by many common virtual machine monitors vdi - A disk format supported by VirtualBox virtual machine monitor and the QEMU emulator iso - An archive format for the data contents of an optical disc (e.g. CDROM). qcow2 - A disk format supported by the QEMU emulator that can expand dynamically and supports Copy on Write aki - This indicates what is stored in Glance is an Amazon kernel image ari - This indicates what is stored in Glance is an Amazon ramdisk image ami - This indicates what is stored in Glance is an Amazon machine image What do people think of this proposal to combine the two into a single format field? I agree the current disk_format/container_format tuple isn't ideal. There's overlap between the two and at the same time, there are things that can't be expressed with the current selection of valid settings. I do think having two separate fields defining the contents, though. There are basically two things that are relevant: The image type and the container format. The image type can be either of kernel, ramdisk, filesystem, iso9660, disk, or other. The container type can be: raw, cow, qcow, qcow2, vhd, vmdk, vdi or qed (and probably others I've forgotten). Container type is essential in deciding whether the hypervisor in question will be able to take the image and read its contents (i.e. map a block of data in the container to a block of data in the contained image). Image type is essential in deciding what to do with it. I.e. *don't* try to attach a kernel as a filesystem, *don't* try to use an iso9660 image as your kernel, *do* attach iso9660 images as CD's, not as hard drives, *do* accept booting a VM with only a disk image attached, *do* require a kernel if you have a filesystem image rather than a disk image, etc. At the moment, we try to guess the user's intent (if they don't pass a kernel, we just boot the image and hope for the best). This is error prone. aki, ari, and ami have always struck me as odd. If you upload an aki to OpenStack, by the time it actually reaches Glance, it's not an aki anymore. Its image type is kernel and its container format is raw. It's indistinguishable from a raw kernel image uploaded by some other mechanism. Same for ari (ramdisk/raw) and ami (filesystem/raw). If anything, aki/ari/ami might be considered a (single) transport format. Uploading an image to EC2 involves a bundling process where the image in question is split up, signed (and encrypted?), uploaded to S3 along with a manifest and then registered. Upon registration, the signature is verified, the image is decrypted(?), and stitched back together to form a kernel image (or ramdisk or machine image). At this point, any remnants of the manifest and the rest of the bundle are gone. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [GLANCE] Proposal: Combine the container_format and disk_format fields in 2.0 Images API
2011/12/2 Donal Lafferty donal.laffe...@citrix.com: During October I noticed that Microsoft's vhdtool.exe creates VHDs that XenServer can't understand. Boy was that painful. The underlying problem is that some vhd's should be described as VM specific. Can you elaborate on this, please? I don't think I understand what VM specific means. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [GLANCE] Proposal: Combine the container_format and disk_format fields in 2.0 Images API
2011/12/2 Donal Lafferty donal.laffe...@citrix.com: The key in my email was to ask whether MIME-like specialisations were appropriate either for combining characteristics of an image into a single property. E.g. container_type/image_type. The example I provided was image_type/vendor-specific-format That second example came from observing that a VHD produced by VHDTOOL.exe as posted on MSDN produced a file that could not be understood by XenServer. In contrast, Ken Bell's 'DiscUtils' as posted on Codeplex produced a VHD that worked fine. When I spoke to Ken, he mentioned he'd noticed that VHDTOOL.exe generated a slightly different format. Now, I doubt Microsoft would host a tool that didn’t support their format. Therefore, there seems to be a difference of opinion as to what constitutes a VHD. I understand there might be differences in implementations of the various formats. Sometimes, this is due to bugs (common if the format was reverse-engineered) or perhaps different (incompatible) versions of the same format. I don't think the correct way to encode these differences is making it vendor specific. As an example, vmdk's generated by QEmu are different from vmdk's generated by VirtualBox, and both of those are different that vmdk's generated by VMWare (which again generates different vmdk's depending on its version), but the compatibility matrix is complicated. I think all vmdk's from QEMu will work in VirtualBox and VMWare, but VMWare and VirtualBox can certainly both generate vmdk's that QEMu doesn't understand. Some of these differences are due to different versions of the vmdk format being used, and some are due to incomplete implementations of the formats. I simply don't think adding a vendor part to the container type string is going to be a very good way to encode this. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] reviewday
Can you explain what some of the errors mean? For instance, for this merge proposal: https://review.openstack.org/#change,1950 the unit tests passed, but the libvirt tests were Hit by Torpedo and XenServer tests had a Chef client timeout. Am I right to assume that at least the chef client timeout problem is an issue in the test setup rather than indicative of a problem with the merge proposal? I have no idea what Hit by Torpedo means :) -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Proposal for Mark McLoughlin to join nova-core
2011/11/29 Vishvananda Ishaya vishvana...@gmail.com: Mark is maintaining openstack for Fedora and has made some excellent contributions to nova. He has also been very prolific with reviews lately. Lets add him to core and make his reviews count towards potential merges! I'd be delighted to have Mark on the core team. +1 -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Providing packages for stable releases of OpenStack
I think there are two distinct use cases here. To me, the PPA's have always been a QA tool. I wanted people willing to help test OpenStack to be able to do so with as little effort as possible. Building packages per-commit gave us that. It seems incredibly counterintuitive to me that someone who wants to help us verify the stable branches need to jump through more hoops to do so. IMO we should be as least as concerned about the quality of stable updates as anything else. This is why I think we should be offering a PPA with per-commit builds from the stable branch(es). This is completely different from a production PPA. I wouldn't dream of pointing people to the above mentioned PPA for their production environment. If someone wants to offer this outside of (but perhaps in cooperation with) OpenStack, that'd be great. I'd be delighted to see companies taking this on and offering a supported OpenStack distribution, but I don't think this is our job for pretty much all the same reasons Thierry outlines. I propose we start building packages from the stable branches and put them in an appropriately named/labeled PPA, such as nova-core/diablo-qa or nova-core/diablo-not-for-production (or perhaps under openstack-stable-maint). At the same time, I'd like to propose that we limit ourselves to fewer supported versions of Ubuntu (for trunk builds as well as these new, stable branch builds). Specifically: * Most recent LTS * Most recent release (which may or may not be an LTS) * Current development release LTS's would go out of support when the subsequent LTS's first point release is released. Non-LTS's would go out of support a month after the subsequent release is out. This means that right now, we'd build: * Lucid (until 12.04.1 is released (July 2012)) * Oneiric (until May 2012) * Precise (until (probably) July 2014) This gives people ample opportunity to upgrade to the next release and at the same time reduces the amount of releases we need to worry about significantly. I think we'd get a valuable QA tool back and we'd reduce the burden of maintaining the per-commit packages by having fewer distro versions to worry about. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Providing packages for stable releases of OpenStack
2011/11/30 Thierry Carrez thie...@openstack.org: Soren Hansen wrote: I propose we start building packages from the stable branches and put them in an appropriately named/labeled PPA, such as nova-core/diablo-qa or nova-core/diablo-not-for-production (or perhaps under openstack-stable-maint). [...] That would work (and inside the current project). Just two remarks: * Expectations are difficult to control. Even if we use an intimidating name, some people will still expect this to provide more than it actually does. For example, people kept thinking that the 2011.3 release PPA would be updated, while it explicitly said it wouldn't. The reason I want the PPA name to be scary looking is exactly because of the lesson learned from those PPA's. It's easy to miss the disclaimers on Launchpad (and if you happen to find the PPA info somewhere else, there might be no disclaimer at all!). The PPA name is the most obvious place to put this. Only if you're running someone else's script to enable it will you never see it. Some people will still miss it, but I think it's the best we can do. * I don't think that's what Vish and Monty are after -- they specifically mentioned the lack of a production-ready distribution channel as the problem that we needed to solve Right. I agree we shouldn't do that. Someone else should. But I don't want that to hold back the creation of the per-commit PPA for diablo/stable which I find important for QA purposes. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Providing packages for stable releases of OpenStack
2011/11/30 Mark McLoughlin mar...@redhat.com: On Wed, 2011-11-30 at 13:07 +0100, Soren Hansen wrote: I propose we start building packages from the stable branches and put them in an appropriately named/labeled PPA, such as nova-core/diablo-qa or nova-core/diablo-not-for-production (or perhaps under openstack-stable-maint). I'm not convinced that distribution specific packaging is the best way to go about this. That's a valid discussion. At the moment, this is what we do for trunk commits. This is how we generally propose that people test things out. I don't see any reason why the mechanics for testing the stable branches should be different. So, a discussion about these mechanisms shouldn't be isolated to the context of the stable branch. I want Fedora users to be able to test out, and get involved with, upstream as easily as Ubuntu users are. I'd be happy for us to build Fedora packages as well, fwiw. Same for other distros. The thought of getting into the game of maintaining this upstream packaging for multiple distros, and e.g. having to make sure any dependencies are packaged for these distros ... ugh. Yes, this is a lot of work. This is one of the primary reasons we chose a reference platform to begin with: Being able to focus the efforts and actually succeed rather than trying to do everything and fail. We had (and have) people involved in the project that could actually take this on. If someone wants to do the same for Fedora (and other distros), that'd be awesome. I don't have anything concrete to offer as an alternative, but I'd love to see something like devstack that runs either from git or tarballs and supports multiple distributions. For production, we recommend people use packages. I think there's a lot of value in using the same installation mechanism for QA as for production. There's also the likes of jhbuild, GARGNOME, minuteman and surely more - perhaps we can take a leaf out of their books? I hope I'n not stepping on anyone's toes, but I consider those things to be relics from a time before things such as PPA's became prevalent. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Database stuff
Hi, guys.
Gosh, this turned out to be long. Sorry about that.
I'm adding some tests for the DB api, and I've stumbled across something
that we should probably discuss.
First of all, most (if not all) of the various *_create methods we have,
return quite amputated objects. Any attempt to access related objects
will fail with the much too familiar:
DetachedInstanceError: Parent instance Whatever at 0x4f5c8d0 is not
bound to a Session; lazy load operation of attribute 'other_things'
cannot proceed
Also, with the SQLAlchemy driver, this test would pass:
network = db.network_create(ctxt, {})
network = db.network_get(ctxt, network['id'])
instance = db.instance_create(ctxt, {})
self.assertEquals(len(network['virtual_interfaces']), 0)
db.virtual_interface_create(ctxt, {'network_id': network['id'],
'instance_id': instance['id']}
self.assertEquals(len(network['virtual_interfaces']), 0)
network = db.network_get(ctxt, network['id'])
self.assertEquals(len(network['virtual_interfaces']), 1)
I create a network, pull it out again (as per my comment above), verify
that it has no virtual_interfaces related to it, create a virtual
interface in this network, and check the network's virtual_interfaces
key and finds that it still has length 0. Reloading the network now
reveals the new virtual interface.
SQLAlchemy does support looking these things up on the fly. In fact,
AFAIK, this is its default behaviour. We just override it with
joinedload options, because we don't use scoped sessions.
My fake db driver looks stuff like this up on the fly (so the
assertEquals after the virtual_interface_create will fail with that db
driver).
So my question is this: Should this be
a) looked up on the fly,
b) looked up on first key access and then cached,
c) looked up when the parent object is loaded and then never again,
d) or up to the driver author?
Or should we do away with this stuff altogether? I.e. no more looking up
related objects by way of __getitem__ lookups, and instead only allow
lookups through db methods. So, instead of
network['virtual_interfaces'], you'd always do
db.virtual_interfaces_get_by_network(ctxt, network['id']). Let's call
this option e).
I'm pretty undecided myself. If we go with option e) it becomes clear to
consumers of the DB api when they're pulling out fresh stuff from the DB
and when they're reusing potentially old results. Explicit is better
than implicit, but it'll take quite a bit of work to change this.
If we go with one of options a) through d), my order of preference is
(from most to least preferred): a), d), c), b).
There's value in having a right way and a wrong way to do this. If it's
either-or, it makes testing (as in validation) more awkward. I'd say
it's always possible to do on-the-fly lookups. Overriding __getitem__
and fetching fresh results is pretty simple, and, as mentioned earlier,
I believe this is SQLAlchemy's default behaviour (somebody please
correct me if I'm wrong). Forcing an arbitrary ORM to replicate the
behaviour of b) and c) could be incredibly awkward, and c) is also
complicated because there might be reference loops involved. Also,
reviewing correct use of something where the need for reloads depends on
previous use of your db objects (which might itself be conditional or
happen in called methods) sounds like no fun at all. With d) it's
pretty straightforward: Do you want to be sure to have fresh responses?
Then reload the object. Otherwise, behaviour is undefined. It's
straightforward to explain and review. Option e) is also easy to
explain and do reviews for, btw.
DB objects with options a) through d) will have trouble making it
through a serializer. After dict serialisation, the object isn't going
to change, unresolved related objects will not be resolved, and
prepopulating everything prior to serialisation is out of the question
due to the potential for loops and very big sets of related objects (and
their related objects and their related objects's related objects,
etc.). I think it would be valuable to not have to think a whole lot
about whether a particular db-like object is a real db object or
whether it came in as a JSON object or over the message bus or whatever.
Only option e) will give us that, AFAICS.
It seems I've talked myself into preferring option e). It's too much
work to do on my own, though, and it's going to be disruptive, so we
need to do it real soon. I think it'll be worth it, though.
--
Soren Hansen | http://linux2go.dk/
Ubuntu Developer | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Database stuff
2011/11/29 Vishvananda Ishaya vishvana...@gmail.com: e) is the right solution imho. The only reason joinedloads slipped in is for efficiency reasons. In an ideal world the solution would be: 1) (explicitness) Every object or list of related objects is retrieved with an explicit call: instance = db.instance_get(id) ifaces = db.interfaces_get_by_instance(id) for iface in ifaces: ip = db.fixed_ip_get_by_interface(iface['id']) 2) (efficiency) Queries are perfectly efficient and all joins that will be used are made at once. So the above would be a single db query that joins all instances ifaces and ips. The way I'd attack these expensive-if-done-one-at-a-time-but-dirt-cheap- if-done-as-one-big-query is to have a method in the generic layer that is taylored for this use case. E.g. def instances_get_all_for_network_with_fixed_ip_addresses(): retval = [] for inst in instance_get_all_by_network(): x = inst.copy() x['fixed_ip_addresses'] = [] for ip in fixed_ip_get_by_instance(inst['id']): x['fixed_ip_addresses'].append(ip['address']) retval.append(x) return x And then, in the sqlalchemy driver, I could override that method with one that issues a query with joinedloads and all the rest of it. The intent is explicit, drivers that have no speedier way to achieve this get a free implementation made up of the more primitive methods. fixed_ip_get_by_instace might also have a default implementation that issues a fixed_ip_get_all() and then filters the results. This way, a new driver would be quick to add, and then we could optimize each query as we move along. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Database stuff
2011/11/29 Jay Pipes jaypi...@gmail.com: There's a very good reason this hasn't happened so far: handling highly relational datasets with a non-relational data store is a bad idea. In fact, I seem to remember that is exactly how Nova's data store started out life (*cough* Redis *cough*) To be fair, we're only barely making use of this in our DB implementation. I don't think we do any foreign key checking at all, and deletes (because we don't actually delete anything, we just mark it as deleted) don't cascade, so there are all sort of ways in which our data store could be inconsistent. Besides, we don't really use transactions. I could easily read the same data from two separate nodes, make different (irreconcilable) changes on both nodes, and write them back, and the last one to write simply wins. In short, it seems to me we're not really getting much out of having a relational data store? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
It's been a bit over a week since I started this thread. So far we've agreed that running the test suite is too slow, mostly because there are too many things in there that aren't unit tests. We've also discussed my fake db implementation at length. I think we've generally agreed that it isn't completely insane, so that's moving along nicely. Duncan has taken the first steps needed to split the test suite into unit tests and everything else: https://review.openstack.org/#change,1879 Just one more core +1 needed. Will someone beat me to it? Only time will tell :) Thanks, Duncan! Anything else around unit testing anyone wants to get into The Great Big Plan[tm]? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Database stuff
2011/11/29 Jay Pipes jaypi...@gmail.com: On Tue, Nov 29, 2011 at 2:58 PM, Soren Hansen so...@linux2go.dk wrote: 2011/11/29 Jay Pipes jaypi...@gmail.com: There's a very good reason this hasn't happened so far: handling highly relational datasets with a non-relational data store is a bad idea. In fact, I seem to remember that is exactly how Nova's data store started out life (*cough* Redis *cough*) To be fair, we're only barely making use of this in our DB implementation. I don't think we do any foreign key checking at all, and deletes (because we don't actually delete anything, we just mark it as deleted) don't cascade, so there are all sort of ways in which our data store could be inconsistent. Because the database schema isn't properly protecting against referential integrity failures does not mean the relational database store is a failure itself. I'm not suggesting it's a failure at all. Besides, we don't really use transactions. I could easily read the same data from two separate nodes, make different (irreconcilable) changes on both nodes, and write them back, and the last one to write simply wins. Sure, but using a KV store doesn't solve this problem... I'm not suggesting it will. My point is simply that using a KV store wouldn't lose us anything in that respect. In short, it seems to me we're not really getting much out of having a relational data store? We're getting out of it what we ask of it. We aren't using scoped sessions properly, aren't using transactions properly, and we aren't enforcing referential integrity. But those are choices we've made, not some native deficiency in relational data stores. I didn't mean to suggest that that was the case at all. The point I'm trying (but failing, clearly) to make is that with the way we're using it, we're not reaping the usual benefits from it, and that we'd in fact not lose anything by using a KV store. As soon as someone can demonstrate the performance, scalability, and robustness advantages of rewriting the data layer to use a non-relational data store, I'm all ears. Until that point, I remain unconvinced that the relational database is the source of major bottlenecks. I understand that MySQL (and the other backends supported by SQLAlchemy, too) scales very well. Vertically. I doubt they'll be bottlenecks. Heck, they're even well-understood enough that people have built very decent HA setups using them. I just don't think they're a particularly good fit for a distributed system. You can have a highly available datastore all you want, but I'd sleep better knowing that our data is stored in a distributed system that is designed to handle network partitions well. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Database stuff
2011/11/29 Jay Pipes jaypi...@gmail.com: On Tue, Nov 29, 2011 at 3:43 PM, Soren Hansen so...@linux2go.dk wrote: 2011/11/29 Jay Pipes jaypi...@gmail.com: Besides, we don't really use transactions. I could easily read the same data from two separate nodes, make different (irreconcilable) changes on both nodes, and write them back, and the last one to write simply wins. Sure, but using a KV store doesn't solve this problem... I'm not suggesting it will. My point is simply that using a KV store wouldn't lose us anything in that respect. I see your point. But then again, it comes down to whether we care about referential integrity or transactional safety. ...and right now we have neither (by choice, not by limitations imposed by the data store). Would you not agree? If we don't, then we're just building a distributed system that has unreliable persistent storage built into it, and that, IMHO, is a bigger problem than the as-yet-unproven assertions around scalability of a relational database in a distributed system. (more below) Yes. This is what we have now. And it sucks. As soon as someone can demonstrate the performance, scalability, and robustness advantages of rewriting the data layer to use a non-relational data store, I'm all ears. Until that point, I remain unconvinced that the relational database is the source of major bottlenecks. I understand that MySQL (and the other backends supported by SQLAlchemy, too) scales very well. Vertically. I doubt they'll be bottlenecks. Heck, they're even well-understood enough that people have built very decent HA setups using them. I just don't think they're a particularly good fit for a distributed system. You can have a highly available datastore all you want, but I'd sleep better knowing that our data is stored in a distributed system that is designed to handle network partitions well. I guess I don't understand this. How do you sleep at night TODAY knowing that the data Nova stores in its persistent storage is wide open to referential integrity problems and transactional state inconsistencies? Not very well at all. If I thought everything was in good shape, I woulnd't have bothered with all of this :) What's the point of having a data store that understands network partitions if we don't care enough to protect the integrity of the data we're putting in the data store in the first place? :( None at all. I hope I haven't said anything to suggest otherwise. MySQL simply was not designed to be distributed. Generally speaking, if you do end up in a situation where there's been a network partition and your master is on one side and you have a slave on the other side, a couple of things can happen: 1. You can automatically promote the slave to master, thus letting both sides of the partition keep going. 2. You can leave the slave be and let the entire one side of the partition be in read-only mode. I think the usual case is 1, since MySQL HA setups are usually designed to handle the case where the master dies rather than handling network partitions. Would you agree with this assertion? If both have acted as master, what happens when the the network is joined again? Hell breaks loose, because MySQL wasn't designed for this sort of thing. Something like Riak, on the other hand, is designed to excel for exactly this sort of situation. It makes no attempt to handle these conflicts (unless you explicitly tell it to just let last write win). If there are conflicts, you get to handle it in your application in whatever way makes sense. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Vulnerability Management concerns: negativity count
2011/11/24 Thierry Carrez thie...@openstack.org: This is actually linked to the next section. If you limit the numbers of members in a vulnerability handling team, you create resentment with those members or companies that are not part of it. The phrasing is there to reassure non-members that there is no advantage for being in. Exactly. We're bootstrapping the team and the process. We (as a project) don't necessarily know the people stepping forward to take on a membership of this team, so it's important that the responsibilities (of which there are many) and privileges (of which there are really none) are clear. I see no reason not to be clear about the ground rules up front, and make it explicit that it's not an early warning list. It's a response team. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Vulnerability Management concerns: negativity count
2011/11/24 Lloyd Dewolf lloydost...@gmail.com: A. As my former boss, as of this week, Matt Mullenweg [1] would so often remind us, don't be so negative -- he literally reminded my VIP Services sub-team of that last week -- it's natural when you are deep in the trenches. Instead use Words that Work. [2] This is not marketing material. It's not meant to sell anything or convince anyone of anything. It's supposed to accurately convey what this team is and what it isn't. If you want to rephrase it, knock yourself out, but being unambiguous trumps sounding good. You don't see legislation being rephrased to make it sound better either :) -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
2011/11/23 Sandy Walsh sandy.wa...@rackspace.com: Thanks Soren, I see what you're doing now and it makes perfect sense. It'll be a nice helper class. Cool. My only snipe would be that mox is generic to any library and this fake only gives the benefit to db operations. We have to remember It's a db operation, so I have to do this. It's another method call so I need to do that I think if it more like for db, I don't need to concern myself with test doubles. There's still a bunch of other stuff where that's not true, but for db, it Just Works[tm]. How much effort would it be to make it into a better/more generic mox library? I don't see how that would even be possible? I'm writing a complete db driver, backed by Python dicts rather than sqlalchemy+sqlite. I can't imagine how you'd build something that generally can expose an API and behaviour identical to an arbitrary module, which seems to be what you're suggesting. Ok, that's not entirely true. I can imagine injecting a proxy in front of the real DB driver, record its behaviour, and on subsequent test runs, I'd return canned responses, but I really wouldn't recommend something like that. It's great of getting some insight into how a particular module is used. You can use that information when writing stubs, mocks, fakes, whatever based on it, but I wouldn't rely on being able to just replay its traffic and have everything work. Am I completely misunderstanding you? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
2011/11/23 Sandy Walsh sandy.wa...@rackspace.com:
I understand what you're proposing, but I'm backtracking a little.
(my kingdom for you and a whiteboard in the same room :)
Well, IRC would be a good start. :) I haven't seen you on IRC for days?
I think that you could have a hybrid of your
db.do_something(desired_return_value)
I may be reading too much into this, but this example suggests you're
not following me, to be honest.
db.instance_create is not a method I'm adding. It is an existing
method. It's the method you use to add an instance to the data store,
so it's not so much about passing it desired return values. It's
about adding an instance to the database in the exactly same fashion
as production could would have done it, thus allowing subsequent calls
to instance_get (or any of the ~30 other methods that return one or
more Instance objects) to return it appropriately. And by
appropriately, I mean: with all the same attributes as one of the real
db drivers would have returned.
to produce:
self.sorens_mox.Set(nova.db, 'instance_get_by_uuid', {'name': 'this or that',
'instance_type_id': 42})
We have this functionality today. My example:
self.stubs.Set(nova.db, 'instance_get_by_uuid', fake_instance_get)
was copied from one of our existing tests.
--
Soren Hansen | http://linux2go.dk/
Ubuntu Developer | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
Ok, this seems like a good time to repeat what I posted to nova-database the other day. tl;dr: I'm adding a fake DB driver as well as a DB test suite that we can run against any of the backends to verify that they act the same. This should address all the concerns I've heard so far. Hi. I just want to let you know that I'm working on a fake DB driver. The two primary goals are to reduce the time it takes to run the test suite (my results so far are very impressive) and simply to have another, independent DB implementation. Once I'm done, I'll start adding tests for it all, and finally, I'll take a stab at adding an alternative, real DB backend. In case you're wondering why I don't write the tests first, it's simply because I don't know how all these things are supposed to work. I hope to have a much better understanding of this once I've written the fake DB driver, and then I'll add a generic test suite that should be able to validate any DB backend. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
2011/11/22 Sandy Walsh sandy.wa...@rackspace.com: I'm not a big fan of faking a database, not only for the reasons outlined already, but because it makes the tests harder to understand. Can you give me an example? I find the opposite to be true, so I'd love to see counterexamples. Most of the time, the data store is not relevant for the tests. I just need to stick an instance into the db, do some stuff, and verify that I get the correct (direct and indirect) output. I don't see how having a mocked db.instance_get is any more readable than a db.instance_create() (or a parameterised create_instance utility method for testing purposes or whatever). I much prefer to mock the db call on a per-unit-test basis so you can see everything you need in a single file. Yes, this could mean some duplication across test suites. But that is better than changes to the fake busting some other test that has different assumptions. That's why I'm adding tests for the fake. To make sure that the fake and the real db drivers act the same. Are we testing the code or are we testing the fake? The code. We have *other* tests for the fake. (real) Unit tests are our documentation and having to jump around to find out how the plumbing works doesn't make for good documentation. I agree. That's exactly why I *don't* want mocks (for this) in the unit tests. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
2011/11/22 Sandy Walsh sandy.wa...@rackspace.com: I suppose there is a matter of preference here. I prefer to look in the setup() and teardown() methods of my test suite to find out how everything hangs together. Otherwise I have to check nova.TestCase when things break. The closer my test can stay to my expectations from unittest.TestCase the happier I am. Sorry, I don't follow. The unit tests would use the fake db driver by default. No per-test-specific setup necessary. Creating the instance in the fake DB would happen explicitly in the individual tests (by way of either calling db.instance_create directly, or by way of some utility function). I can't comment on your fake db implementation, but my fear is this scenario: Test1 assumes db.create_foo() will return 123 and Test2 assumes it will return abc. How do they both comfortably co-exist? And whatever the mechanism, why is it better than just stubs.Set(db.create_foo, _my_create_foo)? I'm confused. That's *exactly* what I want to avoid. By everything sharing the same fake db driver, you can never have one mock that returns one style of response, and another mock that returns another style of response. It's local and it makes sense in the context of that file. But it has to make sense globally. If something you're testing only ever sees an Instance object with a couple of hardcoded attributes on it, because that's what its mock gives it, you'll never know if it'll fail if it gets a more complete, real Instance object. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
2011/11/22 Soren Hansen so...@linux2go.dk: (real) Unit tests are our documentation and having to jump around to find out how the plumbing works doesn't make for good documentation. I agree. That's exactly why I *don't* want mocks (for this) in the unit tests. It's been pointed out to me that I'm not making sense here. Thanks, Devin. :) The point is that I don't want to deal with the plumbing *at all*. I just want to be able to stick stuff in the data store and then let my production (as in non-test) code be able to pull it out again without mucking about with mocks (har har). -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
2011/11/22 Mark Washenberger mark.washenber...@rackspace.com: I think we all agree that we have a mess with the database stubbing that is going on. And I'm confident that the db fake would make that mess more manageable. But the way I see the mess, it comes from having a giant flat db interface and really large testcases. Those are definitely problems, too. The lack of an alternative DB driver (be it a fake or a real one) hasn't helped, either :( Also, by having such large test cases, the generality complexity of setUp() as well as its distance from any given test method increase tremendously. I agree completely. All of this makes it harder to understand the impact of any one line in setUp and nearly impossible to tell which parts of setUp are relevant to a given test. Absolutely. At the moment, it's extremely difficult to tell which parts of the setUp is actually needed for each individual test. But, I have to admit that I'm not backing up the way I think we could make the best improvements with any code at the moment--and Soren is. So I'm not sure I really want to stand in the way. :) I'm hoping the code and the changes it will bring along will speak for itself once I'm done with it. I've implemented 138 of 291 db api methods in my fake driver. I don't know if I'll need all 291 for the unit tests to pass. I hope so, but I'm not sure at all. My only detraction from a big fake is that it in general has to be a pretty faithful implementation of the real interface. Again, with my fake db driver, I'll submit a test suite for it. A test suite that we can run against *any* db driver (real, fake, whatever) to verify that they're all faithful. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
2011/11/22 Sandy Walsh sandy.wa...@rackspace.com:
I suspect the problem is coming in with our definition of unit
tests. I don't think a unit test should be calling out of the method
being tested at all. So anything beyond stubbing out the methods
within the method being tested seems like noise to me. What you're
describing sounds more like integration tests.
If I'm testing a method that includes a call to the db api, the strategy
with which I choose to replace that call with a double does not change
whether the test is a unit test or not.
I'm simply replacing this:
def test_something(self):
self.mox.StubOutWithMock(db, 'instance_get')
db.instance_get(mox.IgnoreArg(), mox.IgnoreArg()
).AndReturn({'name': 'this or that',
'instance_type_id': 42})
exercise_the_routine_that_will_eventually_do_an_instance_get()
verify_that_the_system_is_now_in_the_desired_state()
or this:
def test_something(self):
def fake_instance_get(context, instance_uuid):
return {'name': 'this or that',
'instance_type_id': 42}
self.stubs.Set(nova.db, 'instance_get_by_uuid', fake_instance_get)
exercise_the_routine_that_will_eventually_do_an_instance_get()
verify_that_the_system_is_now_in_the_desired_state()
with this:
def test_something(self):
ctxt = _get_context()
db.instance_create(ctxt, {'name': 'this or that',
'instance_type_id': 42})
exercise_the_routine_that_will_eventually_do_an_instance_get()
verify_that_the_system_is_now_in_the_desired_state()
Not only is this -- to my eye -- much more readable, but because the
fake db driver has been proven (by the db test suite) to give responses
that are exactly like what the real db driver would return, we have
better confidence in the output of the test. E.g. if the real db driver
always sets a particular attribute to a particular default value, it's
remarkably easy to forget to follow suit in an ad-hoc mock, and it's
even easier to forget to update the countless ad-hoc mocks later on, if
such a new attribute is added. This may or may not affect the tested
code's behaviour, but if that was easy to see/predict, we wouldn't need
tests to begin with :)
Over the course of this thread, I've heard many people raise concerns
about whether we'd really be testing the fake or testing the thing that
depends on the fake. I just don't get that at all. Surely a fake DB
driver that is proven to be true to its real counterpart should make us
*more* sure that we're testing our code correctly than an ad-hoc mock
whose correctness is very difficult to verify?
I thought the motive of your thread was to create
fast/small/readable/non-brittle/maintainable tests.
The motive was to gather testing related goals, action items, thoughts,
complaints, whatever. It just so happens that a lot of people (myself
included) think that speeding up the test suite and categorising tests
into true unit tests and everything else are important things to
look at.
Integration tests, while important, make this goal difficult.
I agree. I'm very happy that there's a lot of people doing a lot of work
on the integration test suite so that I can focus more on unit tests. As
I think I've mentioned before, unit tests are really all we can expect
people to run.
So, if we're both talking about real unit tests, I don't seen the
benefit of the fake.
Please elaborate (with my above comments in mind).
As for my example of 123 vs abc, that was a bad example. Let me
rephrase ... in one test I may want to have an environment that has no
pre-existing instances in the db. In another test I may want to have
an environment with a hundred instances.
I'd like to understand how configuring the fake for both of these
scenarios will be any easier than just having a stub. It seems like an
unnecessary abstraction.
First of all, the DB is blown away between each individual test, so we
don't have to worry about its initial state.
In the first scenario, I'd do nothing. I have a clean slate, so I'm good
to go. In the second scenario, I'd just do 100 calls to
db.instance_create. With the mock approach, I'd write a custom
instance_get with a 100 if/elif clauses, returning whatever makes sense
for the given instance_id. Mind you, the objects that I return from my
mock may or may not be valid Instance objects. I can only hope that they
are close enough.
--
Soren Hansen | http://linux2go.dk/
Ubuntu Developer | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
[Openstack] [nova-testing] Efforts for Essex
Hi, guys. We're scattered across enough different timezones to make real-time communication really awkward, so let's see if we can get by using e-mail instead. A good test suite will let you keep up the pace of development. It will offer confidence that your change didn't break any expectations, and will help you understand how things are supposed to work, etc. A bad test suite, on the other hand, will actually do the opposite, so the quality of the unit test suite is incredibly important to the overall health of the project. Integration tests are important as well, but unit tests all we can expect people to run on a regular basis. I'd like to start a bit of discussion around efforts around unit testing for Essex. Think of it as brainstorming. Input can be anything from small, actionable items, to broad ideas, to measurable goals, random thoughts etc. Anything goes. At some point, we can distill this input to a set of common themes, set goals, define action items, etc. A few things from the back of my mind to get the discussion going: = Speed up the test suite = A slow test suite gets run much less frequently than a fast one. Currently, when wrapped in eatmydata, a complete test run takes more than 6 minutes. Goal: We should get that down to less than one minute. = Review of existing tests = Our current tests have a lot of problems: * They overlap (multiple tests effectively testing the same code), * They're hard to understand. Not only are their intent not always clear, but it's often hard to tell how they're doing it. * They're slow. * They're interdependent. The failure of one test often cascades and makes others fail, too. * They're riddled with duplicated code. I think it would be great if we could come up with some guidelines for good tests and then go through the existing tests and highlight where they violate these guidelines. = Test coverage = We should increase test coverage. Adding tests to legacy code is hard. Generally, it's much easier to write tests for new production code as you go along. The two primary reasons are: * If you're writing tests and production code at the same time (or perhaps even writing the tests first), the code will almost automatically designed to be easily tested. * You (hopefully1) know how the code is supposed to work. This is not always obvious for existing code (often written by someone else). Therefore, the most approachable strategy for increasing test coverage is simply to ensure that any new code added is accompanied by tests, but of course new tests for currently untested code is fantastically welcome. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [nova-testing] Efforts for Essex
2011/11/21 Duncan McGreggor dun...@dreamhost.com: Soren, I'm delighted to see this email come across the list -- thanks for jump-starting such a discussion. It probably goes without saying that I completely agree with your assessments above :-) Great! :) The one thing that has bothered me most about the OpenStack unit tests has been that many of them are not really unit tests; the presence of functional or integration testing precludes that possibility. Very true. Do you think we should spend some time splitting our tests into true unit tests and all the rest? That way, we can easily choose to only run unit tests or the whole thing. To counter up-front any objections to being too much of a purist or drinking too much of the TDD koolaid, let me say that I do believe in testing against real databases, across real networks, etc., but those should be tests that can be run optionally and explicitly stated as being integration tests or functional tests. I think it depends on the type of dependency. Some things should definitely not be involved when running unit tests (sqlalchemy+sqlite for instance), but other things are perfectly fine IMO. E.g. a function that validates whether something is a valid IP address may have started its life as a homegrown routine, but eventually gotten replaced by a routine that uses python-ipy or python-netaddr. That means the test suite has a hard dependency on python-ipy or python-netaddr, but I still think this falls squarely in unit testing territory. By using mocked objects (e.g., Michael Foord's mock library), unit test speeds will go WAY up. We use python-mox quite a bit and have half a bajillion ad-hoc mock implementations of various things. I'm actually on a mission to replace many of these with more thorough fake objects. I have a fake db implementation I've been working on for a couple of days that gets me half way through the test suite in less than a minute. It's supposed to eventually replace the countless ad-hoc mock db implementations scattered across our test suite. They were great when they were added, but IMO they're much more a liability now, because they're impossible to maintain. If we could separate out everything that currently hits other systems in the OpenStack amalgam so that they use a separate runner (integration_tests.sh, functional_tests.sh, whatever), and just keep the unit tests in the main test runner, not only will things run much more rapidly, but it will be a no-brainer for folks to start running them compulsively (before submitting patches, before submitting bugs, before asking for help on IRC, etc.). Sounds like a great idea. As for the duplicate code, it's probably fairly obvious to everyone that we could build up a subpackage for testing, that pulls together all the commonly used code and simply import as necessary (as base classes -- OOP -- or instantiate as attribtues -- AOP). That would be fantastic. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Proposal to add Johannes Erdfelt to nova-core
We now have a sufficient amount of +1 to go ahead with this. If noone objects before Tuesday next week, I'll make it so. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Proposal to add Kevin Mitchell to nova-core
That brings us to +5. If noone objects by Tuesday next week, I'll make it so. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Hardware HA
2011/11/10 Viacheslav Biriukov v.v.biriu...@gmail.com: Hi all. What are the best practices for HA of the hardware compute-node, and virtual machines. After googling I found matahari, pacemaker-cloud, but nothing about build-in fiches openstack. 1) How do you create such environments? 2) Does it is right way to use pacemaker-cloud with openstack? Is it stable? I'd avoid depending on anything like that altogether. Try to design your application so that it doesn't depend on any one instance being up. It'll work out better in the long run. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Hardware HA
2011/11/10 Viacheslav Biriukov v.v.biriu...@gmail.com: Hm If we planning vm hosting we work on the other level. So if hw node fails we need fast automatic migration to other node. That's the whole point. For most interesting applications, fast automatic migration isn't anywhere near fast enough. Don't try to avoid failure. Expect it and design around it. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Hardware HA
2011/11/10 Ryan Lane rl...@wikimedia.org: That's the whole point. For most interesting applications, fast automatic migration isn't anywhere near fast enough. Don't try to avoid failure. Expect it and design around it. This assumes all application designers are doing this. Most web applications do this fairly well, but most enterprise applications do this very poorly. I know. That's what makes them a poor fit for the cloud. Hardware HA is useful for more than just poorly designed applications though. I have a cloud instance that runs my personal website. I don't want to pay for two (or more, realistically) instances just to ensure that if my host dies that my site will continue to run. My provider should automatically detect the hardware failure and re-launch my instance on another piece of hardware; it should also notify me that it happened, but that's a different story ;). I'm not sure I count that as High Availability. It's more like Eventual Availability. :) -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Bug fixes and test cases submitted against stable/diablo
2011/11/9 Nachi Ueno ueno.na...@nttdata-agilenet.com: I understand your point. Stop QAing stable/diablo and focus on Essex. Oh, no no. That's not the point. I'm thrilled to have you work on QAing Diablo. The only issue is that the fixes you come up with should be pushed to Essex first. There are two reasons for this: * If we don't push the fixes to Essex, the problems will still be present in Essex and every release after that. * Having them in Essex lets us try them out, vet them and validate them more thoroughly before we let them into the stable branch. When a patch lands in the stable branch it has to be well tested already (unless of course Essex has deviated too much, in which case we'll have to accept the risk of getting it into Diablo directly). However the current situation is different. IMO the quality diablo is not ready for real deployment. In the diablo summit, I think we agreed the policy Do not decrease code coverage on merge. But it is not applied through diablo timeframe,and the diablo has small coverage. This is true :( We are struggling with very tight schedule. X( If our contribution is rejected to the stable/diablo, to maintain our own branch is only option for us. And I don't really want to do this. Yes, I would very much like to avoid this as well. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Libguestfs??
2011/11/3 Ilya Alekseyev ilyaalekse...@acm.org: Actually libguestfs is using in GD RHEL port of OpenStack. The reason why this patch wasn't proposed to trunk is unavailable libguestfs packages in official repository (there is no packages in 10.10 for example). To clarify, the patch was rejected because it /unconditionally/ would use libguestfs. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Git repository for packaging
2011/10/21 Monty Taylor mord...@inaugust.com: On 10/21/2011 03:16 AM, Thomas Goirand wrote: Best, IMHO, would be to maintain python-novaclient directly inside the nova project, since this way we can do a version depend. There was a discussion about this at the design summit, and it was decided against for now because this winds up causing problems for python depends in pip-based installs. (there is no idea of a source package creating multiple binary packages in pip, so for people who just need nova client to have to install nova and all of its depends is problematic.) Just because they're in the same repository, they don't have to be the same source package. They could have separate top-level directories (so each their own setup.py and whatnot), but in the same repository. That way, we can keep them separate, but still in sync. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] nova-compute cannot running after one or two days running.
I don't remember when the change to log attempts to grab locks landed. Maybe your version doesn't have that. In that case, my guess is that it's attempting to grab a lock to fiddle with iptables, but something is already holding that lock. There should be a nova directory in /var/lock. Can you provide an ls - l of that dir? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] OpenStack API Versioning Conventions
2011/10/11 George Reese george.re...@enstratus.com: It's wildly inappropriate to equate a thing with its representation. I didn't say I was right in doing so :) It's a discussion that gets philosophical rather quickly: Should we consider a URI to be a reference to a thing or a reference to a representation of a thing? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] OpenStack API Versioning Conventions
2011/10/11 George Reese george.re...@enstratus.com: See EC2 for someone doing it damn well. I've never had to write new code to talk to them unless I want to take advantage of new functionality. Now I'm confused. EC2 includes the API version in the URI, but I thought you were opposed to that? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] OpenStack API Versioning Conventions
2011/10/11 George Reese george.re...@enstratus.com: Versioning should not be included in the URI. It belongs in the headers. A URI should be a persistent reference to a resource. As such, versioning always breaks that persistent reference. I don't follow. If the version is included in the URI, that's got to be a more persistent reference to a resource than a URI whose behaviour differs depending on a header that you have to include? -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Nova DB Connection Pooling
2011/9/26 Monty Taylor mord...@inaugust.com: I'm not a MySQL guru by any means, but can you explain this to me? I've never read anywhere that MySQL doesn't really like having a bunch of unused connection sitting around for long lifecycles. It seems pretty logical to me to have at least 2 persistent connections to the database to avoid being completely blocked on database calls. So I should probably phrase that a little differently - and some of it is a question of scale. 2 connections, yes - 15 probably not. I've not run MySQL at this scale before. How well will it handle a couple of thousand persistent connections? When does the pain start to kick in? Anyways, this cloud stuff is all about *horizontal* scalability. It does seem increasingly odd (to me, at least) to have the architecture include this central datastore that everything needs to connect to, regardless of how well this datastore scales *vertically*. Something like Riak was designed to scale extremely well horizontally (in much the same way as Swift). Using it will require us to rethink our datastore access quite considerably, but the benefit is painless horizontal scalability (and unicorns and ponies, of course). -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Proposal to add Chris Behrens to nova-core
I just added Chris to nova-core. Welcome to the team, Chris! I'll add you to the review rotation next time that schedule gets updated. -- Soren Hansen | http://linux2go.dk/ Ubuntu Developer | http://www.ubuntu.com/ OpenStack Developer | http://www.openstack.org/ ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
