Please see below: On Thu, Oct 13, 2016 at 9:33 PM, Matt Riedemann <mrie...@linux.vnet.ibm.com> wrote: > On 10/13/2016 7:47 PM, Emilien Macchi wrote: >> >> Greetings OpenStack, >> >> == Background >> >> Since the beginning of OpenStack (or almost), devstack has been used >> as a common tool to deploy OpenStack in CI environment. Most of >> OpenStack projects (if not all) that are written in Python use it to >> deploy the different components. >> While devstack became popular and the reference in term of deployment >> tool for continuous integration, devstack doesn't deploy OpenStack in >> production (versus some tools like Kolla, Fuel, TripleO, Juju, etc). >> It means things might (and did) break when deploying OpenStack outside >> devstack, for different reasons. Some examples: >> >> * until recently, SSL was not tested, and I believe some projects >> still don't test with SSL enabled. >> * IPv6 is not tested everywhere. >> * Production scenarios, with HA (HAproxy or/and Pacemaker) are not tested. >> >> My point here, is that devstack has been doing very good job for its >> simplicity (written in bash) and its large adoption by projects to >> make CI, though we might consider adding more coverage to make sure it >> works outside devstack. >> As an example, Puppet OpenStack modules CI is using a devstack-like >> job (with 3 scenarios), called puppet-openstack-integration [1] but we >> also run TripleO and Fuel CI jobs, to increase coverage and give a >> better feedback on testing. >> >> >> == Proposal >> >> This is not about removing devstack! The idea is to add more coverage >> in an iterative way, with some other tools. >> We started some months ago by running TripleO CI jobs in Ironic and >> Heat gates (experimental pipeline) because TripleO is high consumer of >> Ironic and Heat. >> Also, we recently added our TripleO multinode job in Nova experimental >> pipeline (doc here [2]). >> Now, we are moving forward with python-openstackclient and osc-lib. >> >> I started to draft a document about how we could increase coverage in >> different projects: >> >> https://docs.google.com/spreadsheets/d/1bLg-uEGrQXyRZ-FuR6pf1WT4XN0-6MrlfqEShI7xMxg/edit#gid=0 >> >> (feel free to add your project and give your opinion directly in the >> spreadsheet). >> >> The intention here is to discuss with teams interested by such CI >> coverage. We don't want to slow down or break your gate (example with >> TripleO, our jobs are non-voting outside TripleO and take ~45 min); >> but reduce the feedback loop between developers and deployment tools >> used in production. >> We don't expect developers to investigate why new CI jobs would fail >> (ex: a Nova developer to look at TripleO CI job), it would be unfair. >> Just some horizontal communication would be enough, IRC or email, to >> inform that a patch might break a CI job outside devstack. >> I also want to mention that the proposal is not only about TripleO. I >> used this tool in my examples because I'm working on it but obviously >> this proposal should be open to Big Tent projects that also deploy >> OpenStack. >> >> Please give any feedback, and let's make OpenStack testing stronger! >> Thanks for reading so far, >> >> [1] https://github.com/openstack/puppet-openstack-integration >> [2] https://review.openstack.org/#/c/381838/ >> > > I don't really have a problem with wanting to run non-devstack deployment > tool jobs against project changes on-demand (experimental queue job). That's > why I approved the change to add that TripleO job to nova's experimental > queue. > > The experimental queue is only on-demand though, so reviewers have to be > conscious of running it and even then people don't think to check the > results, or a failure might not be obvious as to what caused it (my patch, > or is this job always broken and is thus in the experimental queue, like the > nova-lxc job?). > > For better or worse devstack is at least universally used and it's THE > default thing we point newcomers to when getting started if they want to > quickly and easily get a development environment with a running openstack on > a single-node up and running to kick the tires. I can't say the same for the > plethora of other deployment projects out there like kolla, ansible, salt, > puppet, chef, tripleo/packstack/rdo, fuel, etc, etc. I think that's what's > really caused the lack of universal adoption of anything besides devstack in > our CI environment. And love it or hate it, I think anyone that's been > around for awhile and tries to debug gate failures is at least used to > hacking on devstack and knows how it works to a certain extent. > > Anyway, as I said, I've got no problem with getting some additional optional > non-voting coverage in other projects besides devstack to at least try and > prevent breaking changes. I worry about trying to move various deployment > jobs into the check queue for multiple projects though, as I think that > would put a pretty serious strain on resources for non-voting jobs, which > we'd like to avoid I think. > > My two cents.
Matt, Emilien, there's one more option, run jobs daily and add the output to the openstack health dashboard. example - http://status.openstack.org/openstack-health/#/g/build_name/periodic-ironic-py35-with-oslo-master > -- > > Thanks, > > Matt Riedemann > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: https://twitter.com/dims __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
