On 04/24/2013 12:18 PM, David Kranz wrote: > Monty, I think there are really two issues here: > > 1. How can we include a wider variety of tests (different configs, > multi-node, performance, stress, etc.) within the system that the infra > group maintains? > 2. Once we can run more kinds of tests, which of them are actually in > the gate, and is the answer the same for all projects?
I agree with both of these points. > There was a lot of discussion at the summit about progress on (1) and it > seems you guys have real plans to address it, perhaps not without some > controversy. I think 1 is easier - both in terms of the things I want to drive in the infra systems, and also in terms of things that other people want to drive. It's an open system - so having more people (like bodepd) add more tests that do more things is straightforward and limited only by the resources people are interested in putting in to it > But there was no real consensus about (2). I continue to believe that as > the number of projects and configurations that we can test explodes, we > will need a more principled way to make such decisions and run some > tests as periodic or advisory instead. The more complex and "real" the > testing gets, the harder it > is to make the gate as reliable as it needs to be. 2 is the trick - because being in the gate itself means it's prescriptive and effects other people - and it means that the infra team needs to be able to address it. The way you've framed this issue I think makes something clearer for me that I've been trying to say- so I'm going to try a blunter approach on part of it: Because the infra team has to be able to support it, the infra team needs/has veto power on what goes into the gate. Now - that's a negative statement - but there's a corollary to that: The infra team is open just like all of the other project teams. If gating on a thing is important enough to someone that they actually add enough staff to the project that become members of the infra team so that the infra team as a whole feels appropriately staffed to deal with it, then I don't think it'll be a problem. It's the same as with any other team - if someone wants a thing in nova, they have to pony up the folks to do it. The current team barely has enough resources to handle the current workload. The reason I get irate about parallel testing efforts is that it does nothing to actually increase the bandwidth of the infra team itself. So - to sum up - We should definitely test more things and more combinations of things - We need people to become involved enough to actually take an active hand in accomplishing whichever of those things they find important - We would love more people to become involved enough that they become part of the infra core team. - Same as all the other projects - working with others instead of doing something locally is always harder at first - but also sees a greater long-term benefit for everyone involved. Monty > On 4/24/2013 11:19 AM, Monty Taylor wrote: >> >> On 04/24/2013 11:07 AM, Brian Lamar wrote: >>> On Apr 24, 2013, at 10:50 AM, Dan Prince <[email protected]> wrote: >>> >>>> ----- Original Message ----- >>>>> From: Brian Lamar <[email protected]> >>>>> So the question again becomes how can we get XenServer into the >>>>> gate. >>>> Hi Lamar, >>>> >>>> I understand there is a long term goal to move towards larger scale >>>> testing on bare metal w/ TripleO, etc. I'm very interested in that >>>> myself too. In the short/mid term however perhaps using SmokeStack >>>> (which supports testing multiple/flexible configurations) would >>>> help cover the scenario you described below via pre-gate testing? >>> I knew I was missing something big when I had finished this email. >>> Absolutely. Maybe our first fight really is how we can get SmokeStack >>> (and thus XenServer testing) as part of the gate. >>> >>> My understanding is that the main gripe is that it needs to be easy >>> for people to see what failed and for them to have access to the >>> nodes to troubleshoot failures. Is this what is holding things back? >>> I'd love to get as many specifics as possible so we can work on >>> getting things integrated. This is probably a question for >>> Jim/Monty. >> There are a few issues around this: >> >> a) puppet/packages in the gate is a HUGE issue that we do not have a >> solution for yet >> b) reliability engineering - running the amount of tests that we run >> with the other system is a really big undertaking and there is a full >> time team of people working basically just on making it work. >> c) duplication of effort. This was the thing that made me yell at Dan >> originally - although I think we're friends now. But we don't want two >> things in the gate driving testing. We have zuul, and we have jenkins, >> and we add new features to them constantly to keep up with the needs of >> the gate. We don't want to add a second parallel system to a collection >> of tools that's already a huge undertaking to manage, because it would >> mean that everything we do to the zuul system would have to be >> duplicated in the non-zuul system. >> >> Now don't get me wrong - I think smokestack has been a very useful tool >> for the community and I think it's great that Dan has provided that >> service for the community. But if we were to "integrate" it in to the >> gate, I'd be much more interested in porting the things that it runs on >> machines to the system that the OpenStack project already runs. >> >> That still doesn't address the issues with adding packaging and config >> management to the gate though. >> >> If smokestack is useful to you to get started on this task, then by all >> means use it! But in terms of gating, I'd really like for us to engage >> in the hard work of meeting all of the requirements that are out there >> in a way that's integrated with the work that's already being done. >> >>>>> I imagine it going something like this: >>>>> >>>>> >>>>> 1) Hardware is provided with XenServer 6.1 installed >>>> SmokeStack currently runs a single node (all in one) XenServer 5.6 >>>> test on upstream branches and comments on them (pre-gate testing). >>>> It does this using 4 Dell R710's (provided by RAX). Why not upgrade >>>> these machines to XenServer 6.1 to cover this? >>> Sounds good to me. I think I should be able to handle this. I'm going >>> to send some emails internally to get this process kicked off. If I >>> don't reply with a timeline in a week can you ping me again? >>> >>>>> 2) Script for Jenkins will boot XenServer VM on hardware >>>>> provided >>>> SmokeStack uses this approach as well (via something called >>>> Kytoon). >>>> >>>>> 3) VM will be configured using...devstack? >>>> SmokeStack uses Puppet w/ Fedora packages I maintain :). I >>>> understand people have various opinions on how to deploy things... >>>> but if it works w/ packages I would argue that is probably "close >>>> enough" to the test coverage you are looking for. The actual >>>> configuration that is used for XenServer testing is easily >>>> modifiable though so long as the puppet modules support it. >>> Agreed. >>> >>>>> 4) This script will be an unofficial gate (aka 3rd party gate) >>>>> until it is considered stable 5) This script will be integrated >>>>> into the official process either as a gate-gate or a >>>>> periodic-gate >>>> SmokeStack comments on reviews pre-commit (via Bellows). It isn't a >>>> gate (that ship sailed at the Diablo conference) but it is a well >>>> defined 3rd party review tool ATM. >>>> >>>> ---- >>>> >>>> Not trying to derail this into a conversation about SmokeStack. But >>>> it does seem like it wouldn't take too much effort to be able to >>>> cover your concerns in the short/mid term via something that is >>>> already running and commenting on reviews upstream. >>> Yup! My sincere apologies for forgetting about SmokeStack. I've used >>> it enough to know that it has caught tons of stuff for us in the >>> past. I very much would like to work with you to at least get >>> XenServer gating. Once again if we can get an enumerated list of >>> apprehensions from others it will go a long way towards success. >> I hope I was clear above on apprehensions above. If not, PLEASE loop >> back with me - it's not my intent to be negative or snarky or anything >> here. >> >> Monty >> >> _______________________________________________ >> OpenStack-Infra mailing list >> [email protected] >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra > > > _______________________________________________ > OpenStack-Infra mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra > _______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
