Very good Francis, you've nailed it and thank you for clearing that up!
On Tue, Jan 7, 2014 at 1:13 PM, Francis Ginther < [email protected]> wrote: > Vincent, > > I think what Chris and Thomi are asking for is a job that aggregates > all of the current jobs we already have that make use of autopilot and > allow them to run with a version of autopilot under test. > > For example, we run autopilot tests for gallery-app, camera-app and > address-book-app, etc. The goal as I understand it is to just re-use > these existing jobs for the purpose of testing the current app with > the new autopilot. My understanding is that running these app test > suites is what Chris and company are doing manually. So the > application tests and jobs are already defined, the request is to > re-purpose them for testing autopilot. > > Autopilot already has significant unit and integration tests that run > on every MP. I think the goal here is to catch any surprises that can > occur when the apps are tested against a new autopilot. > > > So the questions are: Can we have this please? and what are the steps > > needed to make it happen? > > Regarding the actual implementation of doing this: If I understand > this correctly, we already have all of the pieces to do this laying > around. We may be able to cover desktop and touch testing for the > canonical apps via a single parent job that would aggregate all of our > upstream-merger jobs. The mega-job may also work here to provide > 'smoke'-like testing of the touch-image. I think we can achieve > 75%-90% of this fairly easily, but I can't make any promises on how > soon this could be done without more info. > > Chris and Thomi, I'll try to catch you on IRC today to get more details. > > Francis > > > On Tue, Jan 7, 2014 at 2:49 AM, Vincent Ladeuil <[email protected]> > wrote: > >>>>>> Chris Lee <[email protected]> writes: > > > > > Hi All, > > > We (the Tools and Trust team, CC'd in this email) have a request > for a > > > special Jenkins job that will assist in making Autopilot releases > as > > > painless and speedy as possible. > > > > This sounds like a big task. > > > > > Currently for a release of Autopilot to happen there is a lot of > > > manual testing done which takes a long time (a day or so) for > someone > > > to do. > > > > To achieve this big task, I'd say you should start by automating your > > manual testing one job at a time. > > > > Get it running locally and *then* define the jenkins job that should run > > automatically (preferably with a single command defined by the project). > > > > > This testing is effectively a collection of the automated test jobs > > > that are already exist. > > > > You kind of lost me here, either they are automated or they are manual, > > what is the bit I'm missing ? > > > > > Our idea is to create one massive Jenkins job that contains all > (or a > > > subset?) of the existing automated acceptance tests across the > board > > > (including Click, Touch, Desktop etc.) that can be used to > green-light > > > an Autopilot release. > > > > A massive job is likely to fail exponentially if any of the existing jobs > > is not perfectly isolated and will be a real pain to diagnose. > > > > Splitting jobs to create some isolation between them may be imperfect > > but at least provide useful results. > > > > > This job won't be run very often (only when we intend to release) > > > > "Release what you've tested instead of testing what you've released[1]" > > is my motto. > > > > Delaying feedback on new failures only make them harder to fix. > > > > I would aim for a daily run if those tests can't be run on every commit, > > but waiting for a new release is probably too long. > > > > > so it shouldn't tie up resources and doesn't matter if it takes > > > hours to complete. > > > > > > > Autopilot touches so many different test suites, environments and > form > > > factors (desktop and device) and we need to be confident that a > > > release won't break the build. > > > > Like any library, autopilot should indeed have integration tests > > run. Whether its users do so or (better) autopilot itself anticipates > > and run them itself only impacts the feedback length. > > > > > It is getting harder and taking longer to manually run the tests > > > and can also be a moving target (a contrived example; the CI test > > > runner script using an app-armor command that the manual tester > > > isn't aware of, causing the tester grief) > > > > I'm not sure I understand the issue here, is it that the project using > > autopilot doesn't properly capture the constraints on the test > > environment ? Or is it that this environment is not under the project > > control ? Something else ? > > > > > We're hoping to be able to harness the existing infrastructure and > > > test suites to make this whole process a lot easier. > > > > That, surely, is a important goal of the CI infrastructure. > > > > > So the questions are: Can we have this please? and what are the > steps > > > needed to make it happen? > > > > Three parties seem to be involved here: autopilot, the projects that use > > autopilot and the CI infrastructure. > > > > Since any job run in the CI lab should ultimately be perfectly > > reproducible on any dev machine... I'm strongly tempted to say that the > > CI effort should be to ensure that no specific setup bits (that are not > > properly captured by the project itself) exist in the CI infrastructure. > > > > To get there, I think you should start by making sure you can run all > > the tests you care about locally and automatically, documenting any > > issue blocking such a goal. > > > > Vincent > > > > [1]: Or are about to release which is already better ;) > > > > -- > > Mailing list: https://launchpad.net/~canonical-ci-engineering > > Post to : [email protected] > > Unsubscribe : https://launchpad.net/~canonical-ci-engineering > > More help : https://help.launchpad.net/ListHelp > > > > -- > Francis Ginther > Canonical - Ubuntu Engineering - Continuous Integration Team >
-- Mailing list: https://launchpad.net/~canonical-ci-engineering Post to : [email protected] Unsubscribe : https://launchpad.net/~canonical-ci-engineering More help : https://help.launchpad.net/ListHelp
