On 15/10/2011 12:20, Niels Thykier wrote: > > The primary issue with this suite is that you have to hand-craft *all* > the data needed, which is a pain. I strongly suspect that the long > term implications of this is that tests will end up being very hard to > understand and that people will generally avoid writing tests. >
I guess that's why you added "description" files for (almost) each test? I think it is enough to not forget why we added $some test. > The "second" problem is that the test-suite only checks the heidi > result, so it is cannot tell the difference between "pkg with rc bugs" > and "migrating pkgX would break pkgY". > But we do have (thanks to you) minimal tests for those two situations. So, somehow, they are distinguishable. But, I agree it could be a problem with more complicated tests. > > For now, feel free to give the test suite a spin. Either clone it from > [1] or run it from franck [2]. If you clone it, have a look at the > README (there are no "-h" or "--help"). > I didn't find time to play with it enough yet but I find this project very nice. It helps (some of) us to understand better what britney does in some specific situations, serves as a regression test base and tells new briney implementations what they should implement (at least). I have some minor patches: #1) fixes a typo. #2) makes run-single-test exit with a non zero value when test fails (the change is not pretty but kinda wfm™) Ideally, run-single-test should also tell us why the test failed (maybe by `cat diff`?). #) a bug in britney fixed by Adam recently (that requires disabling compatible mode). But the test passes using the buggy britney… so maybe I did something wrong. Kind regards, -- Mehdi Dogguy مهدي الدڤي http://dogguy.org/
>From 85adbdee956c8cd966c8570c502b8f03200d82d0 Mon Sep 17 00:00:00 2001 From: Mehdi Dogguy <[email protected]> Date: Tue, 18 Oct 2011 11:50:59 +0200 Subject: [PATCH 1/3] Fix typo --- bin/runtests.pl | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/bin/runtests.pl b/bin/runtests.pl index c71e838..4c6b48b 100755 --- a/bin/runtests.pl +++ b/bin/runtests.pl @@ -64,7 +64,7 @@ foreach my $t (@tests) { print "$res\n"; } -print "\nSummery:\n"; +print "\nSummary:\n"; print 'Ran ' . scalar (@tests) . " tests\n"; print "Failed tests: $failed\n"; exit $failed ? 1 : 0; -- 1.7.2.5
>From 9f9adfbf200f8bca7152af4f7fce79da73b09f22 Mon Sep 17 00:00:00 2001 From: Mehdi Dogguy <[email protected]> Date: Tue, 18 Oct 2011 14:05:34 +0200 Subject: [PATCH 2/3] Use exit code returned by method 'run' --- bin/run-single-test.pl | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) diff --git a/bin/run-single-test.pl b/bin/run-single-test.pl index 832ff60..154e615 100755 --- a/bin/run-single-test.pl +++ b/bin/run-single-test.pl @@ -16,7 +16,6 @@ unless ( -d $RUNDIR ) { my $bt = BritneyTest->new ({}, "$RUNDIR/$test", "$TESTSET/$test"); $bt->setup; -$bt->run ($britney); - -exit 0; +my $result = $bt->run ($britney); +exit (1 - $result); -- 1.7.2.5
>From 114a7058cb1aa7ad1643400af332b31bdddcb1d5 Mon Sep 17 00:00:00 2001 From: Mehdi Dogguy <[email protected]> Date: Tue, 18 Oct 2011 11:47:27 +0200 Subject: [PATCH 3/3] Add regression test basic-removed-binary-package --- t/basic-removed-binary-package/description | 10 +++++++++ t/basic-removed-binary-package/expected | 3 ++ .../var/data/testing/Dates | 1 + .../var/data/testing/Hints | 1 + .../var/data/testing/Packages_i386 | 21 ++++++++++++++++++++ .../var/data/testing/Sources | 6 +++++ .../var/data/testing/Urgency | 1 + .../var/data/unstable/Hints | 1 + .../var/data/unstable/Packages_i386 | 13 ++++++++++++ .../var/data/unstable/Sources | 6 +++++ 10 files changed, 63 insertions(+), 0 deletions(-) create mode 100644 t/basic-removed-binary-package/description create mode 100644 t/basic-removed-binary-package/expected create mode 100644 t/basic-removed-binary-package/hints/test-hints create mode 100644 t/basic-removed-binary-package/input/FauxPackages create mode 100644 t/basic-removed-binary-package/var/data/testing-proposed-updates/BugsV create mode 100644 t/basic-removed-binary-package/var/data/testing-proposed-updates/Packages_i386 create mode 100644 t/basic-removed-binary-package/var/data/testing-proposed-updates/Sources create mode 100644 t/basic-removed-binary-package/var/data/testing/BugsV create mode 100644 t/basic-removed-binary-package/var/data/testing/Dates create mode 120000 t/basic-removed-binary-package/var/data/testing/Hints create mode 100644 t/basic-removed-binary-package/var/data/testing/Packages_i386 create mode 100644 t/basic-removed-binary-package/var/data/testing/Sources create mode 100644 t/basic-removed-binary-package/var/data/testing/Urgency create mode 100644 t/basic-removed-binary-package/var/data/unstable/BugsV create mode 120000 t/basic-removed-binary-package/var/data/unstable/Hints create mode 100644 t/basic-removed-binary-package/var/data/unstable/Packages_i386 create mode 100644 t/basic-removed-binary-package/var/data/unstable/Sources diff --git a/t/basic-removed-binary-package/description b/t/basic-removed-binary-package/description new file mode 100644 index 0000000..59e9cdb --- /dev/null +++ b/t/basic-removed-binary-package/description @@ -0,0 +1,10 @@ +Regression test (seen with fbreader) + +This is a regression test. The situation has been experienced with fbreader: +src:fbreader dropped one of its binary packages in sid (say bin:A) but the +other binary package bin:fbreader had a dependency on bin:A in testing only. +This led britney2 to think that fbreader could not be removed without increasing +the uninstallability count, which is false since the newer bin:fbreader dropped +its dependency on bin:A in sid. + +This bug has been fixed in britney2.git:94071b16491e90d78605791a316141b58cc3c6fc diff --git a/t/basic-removed-binary-package/expected b/t/basic-removed-binary-package/expected new file mode 100644 index 0000000..9b0ddc1 --- /dev/null +++ b/t/basic-removed-binary-package/expected @@ -0,0 +1,3 @@ +fbreader 0.10.7dfsg-5 i386 lib +libzltext0.10 0.10.7dfsg-5 i386 lib +fbreader 0.10.7dfsg-5 source lib diff --git a/t/basic-removed-binary-package/hints/test-hints b/t/basic-removed-binary-package/hints/test-hints new file mode 100644 index 0000000..e69de29 diff --git a/t/basic-removed-binary-package/input/FauxPackages b/t/basic-removed-binary-package/input/FauxPackages new file mode 100644 index 0000000..e69de29 diff --git a/t/basic-removed-binary-package/var/data/testing-proposed-updates/BugsV b/t/basic-removed-binary-package/var/data/testing-proposed-updates/BugsV new file mode 100644 index 0000000..e69de29 diff --git a/t/basic-removed-binary-package/var/data/testing-proposed-updates/Packages_i386 b/t/basic-removed-binary-package/var/data/testing-proposed-updates/Packages_i386 new file mode 100644 index 0000000..e69de29 diff --git a/t/basic-removed-binary-package/var/data/testing-proposed-updates/Sources b/t/basic-removed-binary-package/var/data/testing-proposed-updates/Sources new file mode 100644 index 0000000..e69de29 diff --git a/t/basic-removed-binary-package/var/data/testing/BugsV b/t/basic-removed-binary-package/var/data/testing/BugsV new file mode 100644 index 0000000..e69de29 diff --git a/t/basic-removed-binary-package/var/data/testing/Dates b/t/basic-removed-binary-package/var/data/testing/Dates new file mode 100644 index 0000000..da362e6 --- /dev/null +++ b/t/basic-removed-binary-package/var/data/testing/Dates @@ -0,0 +1 @@ +fbreader 0.10.7dfsg-5 15232 diff --git a/t/basic-removed-binary-package/var/data/testing/Hints b/t/basic-removed-binary-package/var/data/testing/Hints new file mode 120000 index 0000000..c641a30 --- /dev/null +++ b/t/basic-removed-binary-package/var/data/testing/Hints @@ -0,0 +1 @@ +../../../hints/ \ No newline at end of file diff --git a/t/basic-removed-binary-package/var/data/testing/Packages_i386 b/t/basic-removed-binary-package/var/data/testing/Packages_i386 new file mode 100644 index 0000000..4ca3cd0 --- /dev/null +++ b/t/basic-removed-binary-package/var/data/testing/Packages_i386 @@ -0,0 +1,21 @@ +Package: fbreader +Architecture: i386 +Depends: libzltext0.10 (>= 0.10.7dfsg-4), libzlui-maemo (>= 0.10.7dfsg-4) +Priority: optional +Section: lib +Version: 0.10.7dfsg-4 + +Package: libzlui-maemo +Architecture: i386 +Depends: libzltext0.10 (= 0.10.7dfsg-4) +Priority: optional +Section: lib +Source: fbreader +Version: 0.10.7dfsg-4 + +Package: libzltext0.10 +Architecture: i386 +Priority: optional +Section: lib +Source: fbreader +Version: 0.10.7dfsg-4 diff --git a/t/basic-removed-binary-package/var/data/testing/Sources b/t/basic-removed-binary-package/var/data/testing/Sources new file mode 100644 index 0000000..331fb80 --- /dev/null +++ b/t/basic-removed-binary-package/var/data/testing/Sources @@ -0,0 +1,6 @@ +Package: fbreader +Binary: fbreader, libzlui-maemo, libzltext0.10 +Maintainer: The R-Team <[email protected]> +Standards-Version: 3.9.2 +Section: lib +Version: 0.10.7dfsg-4 diff --git a/t/basic-removed-binary-package/var/data/testing/Urgency b/t/basic-removed-binary-package/var/data/testing/Urgency new file mode 100644 index 0000000..bb2ff17 --- /dev/null +++ b/t/basic-removed-binary-package/var/data/testing/Urgency @@ -0,0 +1 @@ +fbreader 0.10.7dfsg-4 low diff --git a/t/basic-removed-binary-package/var/data/unstable/BugsV b/t/basic-removed-binary-package/var/data/unstable/BugsV new file mode 100644 index 0000000..e69de29 diff --git a/t/basic-removed-binary-package/var/data/unstable/Hints b/t/basic-removed-binary-package/var/data/unstable/Hints new file mode 120000 index 0000000..c641a30 --- /dev/null +++ b/t/basic-removed-binary-package/var/data/unstable/Hints @@ -0,0 +1 @@ +../../../hints/ \ No newline at end of file diff --git a/t/basic-removed-binary-package/var/data/unstable/Packages_i386 b/t/basic-removed-binary-package/var/data/unstable/Packages_i386 new file mode 100644 index 0000000..905173e --- /dev/null +++ b/t/basic-removed-binary-package/var/data/unstable/Packages_i386 @@ -0,0 +1,13 @@ +Package: fbreader +Architecture: i386 +Depends: libzltext0.10 (>= 0.10.7dfsg-4) +Priority: optional +Section: lib +Version: 0.10.7dfsg-5 + +Package: libzltext0.10 +Architecture: i386 +Priority: optional +Section: lib +Source: fbreader +Version: 0.10.7dfsg-5 diff --git a/t/basic-removed-binary-package/var/data/unstable/Sources b/t/basic-removed-binary-package/var/data/unstable/Sources new file mode 100644 index 0000000..e97ec04 --- /dev/null +++ b/t/basic-removed-binary-package/var/data/unstable/Sources @@ -0,0 +1,6 @@ +Package: fbreader +Binary: fbreader, libzltext0.10 +Maintainer: The R-Team <[email protected]> +Standards-Version: 3.9.2 +Section: lib +Version: 0.10.7dfsg-5 -- 1.7.2.5
