Hi, After a quick hangout, we hashed out a rough first draft:
https://wiki.canonical.com/UbuntuEngineering/CI/HandoffCriteria ...with emphasis on the *rough* part. I'd love feedback on this - what have I forgotten? Cheers, On Fri, Oct 31, 2014 at 10:13 AM, Francis Ginther < [email protected]> wrote: > To your scrum questions. I don't have any experience here either. The > handoff checklist looks a lot like acceptance criteria for the QA story to > add test X, but there's also an expectation that once that acceptance > criteria is met, the CI team will prioritize the next step work to complete > the process. I need to do a little research. > > Also for defining this criteria, I normally would rely on the product > owner and/or scrummaster to be the contact person for this, but since both > are on holiday all week... > > Francis > > On Thu, Oct 30, 2014 at 11:40 AM, Francis Ginther < > [email protected]> wrote: > >> Thomi, >> >> If you're willing to iterate on this a couple times, I think we can >> converge on a set of guidelines rather then try to define the perfect set >> up front. I agree with your first two major starting points: >> >> - All test suites in dep8/autopkgtest format. >> - Results to be stored in a subunit binary stream. >> - ...which is an archived artifact. >> - Tests to be in a launchpad branch (of a specific project? any >> project?) >> >> I think we start with a launchpad branch as that gives access to the dep8 >> tests in a known location. I don't think the project itself matters as long >> as it contains the right set of tests. We want to point the test runner at >> a dep8 test and just say 'go' without having to skip certain tests or >> filter resutls [1]. If the tests can be executed by 'bzr branch ...' and >> 'adt-run ...', then I think this first set of criteria is met. >> >> The third point: >> >> - Evan mentioned that he didn't want to keep running tests in >> jenkins. The tests we'd like to hand over currently run in jenkins. What's >> the proposed alternative? How can we set up those test jobs to prove that >> these suites are stable? >> >> Evan may have additional insight, but at a minimum, we want to get to a >> point where it doesn't matter where the tests are run and therefore tests >> shouldn't rely on a specific jenkins plugin or feature. The touch devices >> are currently available only through jenkins, but the expectation is that >> these will eventually be just adt-run slaves attached to a queue of test >> requests. >> >> Based on our session in DC [2], there a couple of additional bits of >> criteria: >> >> 1) Pushing results to an external service (nfss) and keeping the >> credentials secret. >> - Can we specify that the data to push to nfss be in a consistent >> format within the subunit stream? The idea being that a generic worker >> could extract the data from the stream and push it regardless of what test >> generated the results. Only that worker then needs to have the credentials. >> >> 2) Dealing with potentially destructive tests (e.g. the lrt tests are >> known to put the device in a non-usable state) >> - For the handoff criteria, we would need to know that this is an >> expected outcome of the test and what the completion criteria is. If we >> treat this as an infrastructure failure and just retry the test, it is >> likely to never complete. I don't know a good way to handle this >> generically and perhaps there ultimately isn't one. >> >> 3) Scheduling/Triggering/Monitoring criteria >> - What is the trigger to start a test (i.e. per-image, daily, per-MP, >> etc.) >> - What, if anything, needs to be monitored. I think you already have >> a start here, but we may need to revisit this to find the best location to >> store the alerts. >> >> [1] - We know that the dep8/autopkgtest format does not yet support >> hardware/platform constraints (i.e. this test needs a touch device), but we >> can hopefully keep the need for this minimized for now. >> [2] - >> https://docs.google.com/a/canonical.com/document/d/1LlyWTf7SiPU305uNhJ5H4YH8wOGtTVbbgq6QJ0L6HkE/edit >> >> Francis >> >> On Wed, Oct 29, 2014 at 9:48 PM, Thomi Richards < >> [email protected]> wrote: >> >>> Hello, >>> >>> >>> It's highly likely that goal for the inaugural sprint for the QA >>> projects team will revolve around getting a number of test cases 'up to >>> scratch' to be handed over to the CI team. >>> >>> The problem is that the CI team haven't had time to complete the 'test >>> handover checklist' we discussed in D.C. (as an aside, I'd love to know >>> from the scrum-experts on this list whether there's a standard way to >>> manage dependencies between multiple scrum teams). Additionally, the >>> 'handover' doesn't seem like something that fits within the scrum >>> framework. I'd like some direction from the CI team as to how you see this >>> happening. >>> >>> >>> Regarding the tests themselves, I'd love to get a rough idea of what >>> your requirements are. I have jotted down some thoughts that might help get >>> us started: >>> >>> >>> - All test suites in dep8/autopkgtest format. >>> - Results to be stored in a subunit binary stream. >>> - ...which is an archived artifact. >>> - Tests to be in a launchpad branch (of a specific project? any >>> project?) >>> - Evan mentioned that he didn't want to keep running tests in >>> jenkins. The tests we'd like to hand over currently run in jenkins. >>> What's >>> the proposed alternative? How can we set up those test jobs to prove that >>> these suites are stable? >>> >>> >>> I'd love some guidance on the points above, so the work we produce is >>> aligned with what CI will accept. >>> >>> Please accept my apologies if this seems like me trying to interrupt >>> your sprint. I guess that 's exactly what I'm doing, but you should let me, >>> because... reasons :D >>> >>> Cheers, >>> -- >>> Thomi Richards >>> [email protected] >>> >>> -- >>> Mailing list: https://launchpad.net/~canonical-ci-engineering >>> Post to : [email protected] >>> Unsubscribe : https://launchpad.net/~canonical-ci-engineering >>> More help : https://help.launchpad.net/ListHelp >>> >>> >> >> >> -- >> Francis Ginther >> Canonical - Ubuntu Engineering - Continuous Integration Team >> > > > > -- > Francis Ginther > Canonical - Ubuntu Engineering - Continuous Integration Team > -- Thomi Richards [email protected]
-- Mailing list: https://launchpad.net/~canonical-ci-engineering Post to : [email protected] Unsubscribe : https://launchpad.net/~canonical-ci-engineering More help : https://help.launchpad.net/ListHelp
