(Apologies, this started on the TC list, and really should have started on -dev, correctly posting here now for open discussion)
There were a few chats at summit about this, mostly on the infra / devstack / qa side of the house. Consider the following a straw man to explain the current state of the world, and what I'd like to see change here I call out projects by name here, not to make fun of them, but that I think concrete examples bring the point home much more than abstract arguments (at least for me). This is looking at raising the bar quite a bit along the way. However, as someone that spends a lot of time trying to keep the whole ball of wax holding together, and is spending a lot of time retroactively trying to get projects into our integrated gate (and huge pain to everyone, as their gate commits get bounced by racey projects), I think we really need to up this bar if we want a 20 integrated project version of OpenStack to hold together. ===================================================== Proposed new Incubation and Graduation Requirements ===================================================== The Current State of the World ============================== The integrated gate in OpenStack is a set of devstack / tempest tests which run symmetrically between projects. I.e. we run all the tempest tests (nova, keystone, swift, etc.), on a change to cinder. That means that nova can ensure that a cinder change that would break them will be prevented from landing. An example being the gate-tempest-devstack-vm-full job. OpenStack is currently 9 integrated projects in Havana, 10 once we get to Icehouse (+Trove). Heat added tests to our integrated gate during the Havana cycle, though the deeper guest testing is not part of our gate. Ceilometer has been integrated for 2 releases, and doesn't have integrated gate testing. Trove is just now exploring how they'd get into the gate for Icehouse release. Upgrade testing situation is worse. Only 5 projects currently are part of upgrade testing: Nova, Cinder, Swift, Glance, Keystone. The rest are not set up in that model at all, so we have no infrastructure to ensure the other projects don't break upgrades of themselves. Heat, Trove, and Savana represent an interesting challenge from current gate models in that real validation requires something beyond a trivial guest, as they care about the contents inside compute resources. This desire is likely to grow with other "Layer 4" projects coming into the OpenStack ecosystem [1]. Ironic as an incubated project provides an even different set of challenges, as not only do we not have a gating approach, but we also really should have an approach to test an upgrade from nova-baremetal => ironic to ensure there is a migration path for people in the future. Ceilometer once relied on a version of MongoDB that worked in the gate, but they were never gating, so now they rely on a version of MongoDB that doesn't work in the gate. They made a technical decision to upgrade requirements with no gate feedback because they weren't actually integration testing on a regular basis. Proposed Incubation requirements ================================ Once something becomes an integrated project, it's important that they are able to run in the gate. Both devstack and devstack-gate now support hooks, so with a couple of days of work any project in stackforge could build a gate job which sets up a devstack of their configuration, including their code, running some project specific test they feel is appropriate to ensure they could run in the gate environment. This would ensure an incubated project works with OpenStack global requirements, or if it requires something new, that's known very clearly before incubation. Proposed Graduation requirements ================================ All integrated projects should be in the integrated gate, as this is the only way we provably know that they can all work together, at the same level of requirements, in a consistent way. During incubation landing appropriate tests in Tempest is fair game. So the expectation would be that once a project is incubated they would be able to land tests in tempest. Before integrated we'd need to ensure the project had tests which could take part in the integrated gate, so as soon as a project is voted integrated, it has some working integrated gate tests. (Note: there is actually a symmetric complexity here, to be worked out later). Proposed Stable Release requirements ==================================== We have this automatic transition that happens when a project that's integrated for a release, actually releases as part of that. I.e. Trove and Icehouse. There is no additional TC decision about whether or not Trove is part of the stable release, once integrated, it just is. Nothing that it does over that cycle will kick it out of the stable release. This is one of the reasons it needs to be in the integrated gate **before** graduation. Additionally, upgrade path is critically important to our users, and the number one piece of feedback we received from the User Survey. It was also important enough to our developers that it was scattered all over the Icehouse Design Summit. All integrated projects should be included in upgrade testing the moment they are in a stable release. (ex: when Icehouse is released, Trove should be in master grenade, and upgrade testing from Icehouse -> master for the J cycle from day one). TL;DR ===== - To get to incubated you must have a devstack-gate job - To get to integrated you must have useful tests in the integrated gate - To get into the stable/release you must have upgrade testing enabled Raised Questions ================ - what about existing incubated projects, what would be their time frame to get with this new program - what about existing integrated projects that currently don't exist with either an upgrade or gate story? - what about an upgrade deprecation path (i.e. nova-network => neutron, nova-baremetal => ironic) 1. http://hackstack.org/x/blog/2013/09/05/openstack-seven-layer-dip-as-a-service/ (the layers pattern is useful from a testing requirements model) -- Sean Dague http://dague.net
signature.asc
Description: OpenPGP digital signature
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev