Hi, Adding CI team so they are aware of this, and can chime in if appropriated.
On Thu, Mar 12, 2015 at 4:00 PM, Nicholas Skaggs < [email protected]> wrote: > Greetings all! It's come to my attention recently there is a bit of a > fatal flaw in our testing process for images. I'll be brief and to the > point. As of today we cannot: > > - recreate test results with certainty > - test old images > > The reason for both of these issues is because we grab our test > dependencies on the fly. In doing so, we are at the mercy of the archive > being unchanging. As Max found out yesterday, in practice this means > sometimes you can't even run tests against the latest images because the > archive changed after it was built! It also means, we most certainly cannot > run tests on older images. The archive changes, even for things like the > rtm images which has a released version of ubuntu as it's base. > > So what workarounds exist? Martin has added a nicer error message when you > encounter this [1], and also has supported changing to r/w mode just to > update the apt-indexes [2]. Sometimes this helps, but usually not as the > archive wants to update critical packages that not only change the system > you are attempting to test, but also fail to install. > > So what solutions might be possible? What is needed is a snapshot of the > archive more or less to coincidence with the image. Barring the archive > suddenly deciding to retain old packages, this really isn't possible. So we > could perhaps explore creating and building click packages that contain > tests and their dependencies. As these are dependent on the image > themselves for our purposes they would need to be created (and hosted?) by > CI. Then presumably autopkgtest would have to know where they are and how > to get them. And of course, it's still a broken story for our community who > doesn't have such a service willing to create copies of needed packages at > image build time. Presumably these packages might also be possible to > create at click build time, but they would need to be built at the same > time and then kept. > > Since the aforementioned options are really hacks anyway, there might be > another option. In practice most test dependencies aren't hugely connected > with the platform, so the update hack could work for them. This fails for > autopilot however. If we packaged autopilot and the helpers as a click (or > debian package ::shudder::) that is tied with unity8 and the image, it's > likely we could hack / workaround not being able to test at all. > > So in summary we really can't test old images. And sometimes we can't even > test the current image. That is definitely a problem, but we until now we > have simply retained old results and worked-around / ignored the pain of > sometimes having an untestable image. Since CI hasn't yet adopted using > adt, this issue hasn't yet completely reared it's head > The even greater issue IMHO, is that we can't re-run tests with certainty. > Ideally the image should be read-only, should remain static and we should > have a static testing payload to deploy. The payload should be the same > today as it was 2 weeks ago when we first ran the tests. The results should > be consistent. We can't say that today and that makes the testing we do > suspect. > > > Nicholas > > 1. https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/1425682 > 2. https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/1342838 > > -- > qa-team mailing list > [email protected] > Modify settings or unsubscribe at: https://lists.ubuntu.com/ > mailman/listinfo/qa-team > -- Úrsula Junque Ubuntu CI Engineer [email protected] [email protected] [email protected] Ubuntu - "I am what I am because of who we all are." Linux user #289453 - Ubuntu user #31144
-- Mailing list: https://launchpad.net/~canonical-ci-engineering Post to : [email protected] Unsubscribe : https://launchpad.net/~canonical-ci-engineering More help : https://help.launchpad.net/ListHelp
