Hi Dmitrijs, On 06/11/13 04:30, Dmitrijs Ledkovs wrote:> I've committed a few fixes to fix test-suite errors on PPAs: > > * set & use XDG_RUNTIME_DIR, if not set as otherwise spawning session > init was failing > > * which uncovered that test_umask failure, too restrictive umask under > test (remember session-init needs to create subdirectories) > > * and disabled timing tests, since those don't work on virtualised builders > > So at the moment this brings all daily builds back to green. Let's try > to keep those up =) Thanks! Agreed, but let's not forget that *all* the tests are "green" prior to any push to lp:upstart (having passed locally). But as we know, new tests sometimes fail in the slightly odd lp build environment. Maybe using sbuild-launchpad-chroot would help a little (although that still won't give an identical environment AIUI).
> > WTR declaring failing tests, skipped, expected fail, and bailing out. > > At the moment, we mostly abort and/or skip executing tests > with/without telling the user about it. I think it would be best to > implement TAP[1] output for our tests, such that we can properly skip > tests, or mark them as expecting to fail. > > Announcing that "some tests may or may not fail, because one has > overlayfs mountpoint" is not helpful. Saying that "test_conf test > #3-7" are expected to fail with a reason due to overlayfs mountpoint, > is much better. > > In addition it should enable us to write tests, which are known to > fail at the moment and mark them as TODO for future. > > Similarly this will improve ability to report and compare test-suite > results on different platforms (e.g. how many tests are not run in > virtualised PPAs vs nonvirt vs overlayfs). > > > [1] http://en.wikipedia.org/wiki/Test_Anything_Protocol > Take a close look at https://bugs.launchpad.net/libnih/+bug/933717. I think we should resurrect this and see if Scott would be happy to merge it. I did discuss it a long time ago with Scott and I think his concern was retaining the exiting API. If you look at the attachment, you'll see I've proposed defining NIH_TEST_VERSION_2 to enable the new functionality, but there are other options we could explore. With minimal changes to the existing Upstart tests (basically we just need to run sed over the code) we can do everything you suggest and a whole lot more. That, plus fixing automake to actually show verbose logs by default ('serial-tests' or add some logic to display the verbose log at completion time). James. -- James Hunt ____________________________________ Ubuntu Foundations Team, Canonical.
signature.asc
Description: OpenPGP digital signature
-- upstart-devel mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/upstart-devel
