Bug#806420: debian-edu: test suite times out on ci.debian.net

[Wolfgang Schweer]

On Sun, Mar 18, 2018 at 03:12:34PM +0100, Paul Gevers wrote:
> I would rather have somebody knowlegdeable first judge if really all
> combination need to be run, or if a reduced set is good enough. As I
> believe my fixup will waste needless time on the ci network.

Well, only some knowledge; Petter's feedback appreciated.

While yet another desktop (LXQt having been added for Buster) should be 
added, the number of combinations could be reduced if only KDE is tested 
with all profiles. I figure there'll be enough information if the 
'Workstation' profile gets tested for the remaining desktops.

Patch attached.

Wolfgang
From 8c7296927df5dc43c1bee1858c838b4af1b422da Mon Sep 17 00:00:00 2001
From: Wolfgang Schweer 
Date: Thu, 29 Mar 2018 20:50:34 +0200
Subject: [PATCH 1/2] Add lxqt to the list of desktops to test.

---
 debian/tests/test-tasksel-desktop | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/debian/tests/test-tasksel-desktop b/debian/tests/test-tasksel-desktop
index ab4bac05..843c6cda 100644
--- a/debian/tests/test-tasksel-desktop
+++ b/debian/tests/test-tasksel-desktop
@@ -13,7 +13,7 @@ error() { error=1; echo error: "$@"; }
 echo debian-edu-install debian-edu-install/profile multiselect Workstation \
 | debconf-set-selections
 
-for desktop in kde gnome lxde xfce mate ; do
+for desktop in kde gnome lxde xfce mate lxqt ; do
 echo tasksel tasksel/desktop multiselect "$desktop" \
 	| debconf-set-selections
 wanted="education-desktop-$desktop"
-- 
2.16.3


From 78003dd1b3d93b97cdd9030e4ad67f2e710b752c Mon Sep 17 00:00:00 2001
From: Wolfgang Schweer 
Date: Thu, 29 Mar 2018 20:52:50 +0200
Subject: [PATCH 2/2] Reduce number of test combinations; kde: full range,
 others: only WORKSTATION.

---
 debian/tests/control | 46 +-
 1 file changed, 1 insertion(+), 45 deletions(-)

diff --git a/debian/tests/control b/debian/tests/control
index d651e873..5095fe94 100644
--- a/debian/tests/control
+++ b/debian/tests/control
@@ -30,62 +30,18 @@ Test-Command: PROFILE=Workstation DESKTOP=gnome debian/tests/test-metapkgs
 Depends: education-common, how-can-i-help
 Restrictions: needs-root allow-stderr breaks-testbed
 
-Test-Command: PROFILE=Roaming-Workstation DESKTOP=gnome debian/tests/test-metapkgs
-Depends: education-common, how-can-i-help
-Restrictions: needs-root allow-stderr breaks-testbed
-
-Test-Command: PROFILE=LTSP-Server DESKTOP=gnome debian/tests/test-metapkgs
-Depends: education-common, how-can-i-help
-Restrictions: needs-root allow-stderr breaks-testbed
-
-Test-Command: PROFILE=Standalone DESKTOP=gnome debian/tests/test-metapkgs
-Depends: education-common, how-can-i-help
-Restrictions: needs-root allow-stderr breaks-testbed
-
 Test-Command: PROFILE=Workstation DESKTOP=lxde debian/tests/test-metapkgs
 Depends: education-common, how-can-i-help
 Restrictions: needs-root allow-stderr breaks-testbed
 
-Test-Command: PROFILE=Roaming-Workstation DESKTOP=lxde debian/tests/test-metapkgs
-Depends: education-common, how-can-i-help
-Restrictions: needs-root allow-stderr breaks-testbed
-
-Test-Command: PROFILE=LTSP-Server DESKTOP=lxde debian/tests/test-metapkgs
-Depends: education-common, how-can-i-help
-Restrictions: needs-root allow-stderr breaks-testbed
-
-Test-Command: PROFILE=Standalone DESKTOP=lxde debian/tests/test-metapkgs
-Depends: education-common, how-can-i-help
-Restrictions: needs-root allow-stderr breaks-testbed
-
 Test-Command: PROFILE=Workstation DESKTOP=xfce debian/tests/test-metapkgs
 Depends: education-common, how-can-i-help
 Restrictions: needs-root allow-stderr breaks-testbed
 
-Test-Command: PROFILE=Roaming-Workstation DESKTOP=xfce debian/tests/test-metapkgs
-Depends: education-common, how-can-i-help
-Restrictions: needs-root allow-stderr breaks-testbed
-
-Test-Command: PROFILE=LTSP-Server DESKTOP=xfce debian/tests/test-metapkgs
-Depends: education-common, how-can-i-help
-Restrictions: needs-root allow-stderr breaks-testbed
-
-Test-Command: PROFILE=Standalone DESKTOP=xfce debian/tests/test-metapkgs
-Depends: education-common, how-can-i-help
-Restrictions: needs-root allow-stderr breaks-testbed
-
 Test-Command: PROFILE=Workstation DESKTOP=mate debian/tests/test-metapkgs
 Depends: education-common, how-can-i-help
 Restrictions: needs-root allow-stderr breaks-testbed
 
-Test-Command: PROFILE=Roaming-Workstation DESKTOP=mate debian/tests/test-metapkgs
-Depends: education-common, how-can-i-help
-Restrictions: needs-root allow-stderr breaks-testbed
-
-Test-Command: PROFILE=LTSP-Server DESKTOP=mate debian/tests/test-metapkgs
-Depends: education-common, how-can-i-help
-Restrictions: needs-root allow-stderr breaks-testbed
-
-Test-Command: PROFILE=Standalone DESKTOP=mate debian/tests/test-metapkgs
+Test-Command: PROFILE=Workstation DESKTOP=lxqt debian/tests/test-metapkgs
 Depends: education-common, how-can-i-help
 Restrictions: needs-root allow-stderr breaks-testbed
-- 
2.16.3



signature.asc

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Paul Gevers]

Hi Holger,

On 16-03-18 14:38, Holger Levsen wrote:
> ic, thanks for those explainations and even more for the fix-up! I've merged 
> your commit now, should I upload? :)

I would rather have somebody knowlegdeable first judge if really all
combination need to be run, or if a reduced set is good enough. As I
believe my fixup will waste needless time on the ci network.

Paul



signature.asc
Description: OpenPGP digital signature

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Holger Levsen]

On Fri, Mar 16, 2018 at 01:13:00PM +0100, Paul Gevers wrote:
> I checked the first results and noticed that I send the patch from an
> intermediate git commit of mine, instead of the final one.

ah :/

> My whole discussion about large combination set was for the test suite
> with the attached patch applied.
> 
> As you can see, there seems to be quite a bit of redundancy. I propose
> to first fix the issue with the current test (it appears something
> genuine in debian-edu) before adding more tests.
> 
> (The breaks-testbed is to make sure every tests is started with a
> pristine container as discussed in the #debci IRC channel yesterday).

ic, thanks for those explainations and even more for the fix-up! I've merged 
your commit now, should I upload? :)


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Paul Gevers]

Arggg,

On 16-03-18 12:49, Holger Levsen wrote:
> Hi Paul,
> 
> On Fri, Mar 16, 2018 at 06:11:57AM +0100, Paul Gevers wrote:
>> As an encouragement for you to upload soon with my patch (or an improved
>> one skipping lots of tests), Antonio removed the blacklisting
>> yesterday¹, which is live as we speak.
> 
> Antonio removed the blacklisting yesterday, because yesterday I uploaded
> src:debian-edu with your patch included :)

I checked the first results and noticed that I send the patch from an
intermediate git commit of mine, instead of the final one.

My whole discussion about large combination set was for the test suite
with the attached patch applied.

As you can see, there seems to be quite a bit of redundancy. I propose
to first fix the issue with the current test (it appears something
genuine in debian-edu) before adding more tests.

(The breaks-testbed is to make sure every tests is started with a
pristine container as discussed in the #debci IRC channel yesterday).

Paul
From a4570e8765e017e41deaa6da3418b2fd637282bf Mon Sep 17 00:00:00 2001
From: Paul Gevers 
Date: Thu, 15 Mar 2018 13:26:10 +0100
Subject: [PATCH] Add missing combinations from the original test suite

---
 debian/tests/control | 86 ++--
 1 file changed, 83 insertions(+), 3 deletions(-)

diff --git a/debian/tests/control b/debian/tests/control
index d1611a5a..d651e873 100644
--- a/debian/tests/control
+++ b/debian/tests/control
@@ -1,11 +1,91 @@
 Tests: test-tasksel-desktop
 Depends: education-common
-Restrictions: needs-root allow-stderr
+Restrictions: needs-root allow-stderr breaks-testbed
 
 Test-Command: PROFILE=Minimal debian/tests/test-metapkgs
 Depends: education-common, how-can-i-help
-Restrictions: needs-root allow-stderr
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=Main-Server debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
 
 Test-Command: PROFILE=Workstation DESKTOP=kde debian/tests/test-metapkgs
 Depends: education-common, how-can-i-help
-Restrictions: needs-root allow-stderr
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=Roaming-Workstation DESKTOP=kde debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=LTSP-Server DESKTOP=kde debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=Standalone DESKTOP=kde debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=Workstation DESKTOP=gnome debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=Roaming-Workstation DESKTOP=gnome debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=LTSP-Server DESKTOP=gnome debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=Standalone DESKTOP=gnome debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=Workstation DESKTOP=lxde debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=Roaming-Workstation DESKTOP=lxde debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=LTSP-Server DESKTOP=lxde debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=Standalone DESKTOP=lxde debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=Workstation DESKTOP=xfce debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=Roaming-Workstation DESKTOP=xfce debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=LTSP-Server DESKTOP=xfce debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=Standalone DESKTOP=xfce debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr breaks-testbed
+
+Test-Command: PROFILE=Workstation DESKTOP=mate debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Paul Gevers]

Hi Holger,

On 16-03-18 12:49, Holger Levsen wrote:
> On Fri, Mar 16, 2018 at 06:11:57AM +0100, Paul Gevers wrote:
>> As an encouragement for you to upload soon with my patch (or an improved
>> one skipping lots of tests), Antonio removed the blacklisting
>> yesterday¹, which is live as we speak.
> 
> Antonio removed the blacklisting yesterday, because yesterday I uploaded
> src:debian-edu with your patch included :)

I only checked the state of this bug. It is still open. Did you forget
to close it, or is that on purpose?

Paul



signature.asc
Description: OpenPGP digital signature

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Holger Levsen]

Hi Paul,

On Fri, Mar 16, 2018 at 06:11:57AM +0100, Paul Gevers wrote:
> As an encouragement for you to upload soon with my patch (or an improved
> one skipping lots of tests), Antonio removed the blacklisting
> yesterday¹, which is live as we speak.

Antonio removed the blacklisting yesterday, because yesterday I uploaded
src:debian-edu with your patch included :)


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Paul Gevers]

Hi,

On Fri, 27 Nov 2015 11:44:35 +0100 Petter Reinholdtsen 
wrote:
> [Antonio Terceiro]
> > I am blacklisting debian-edu for now, and will revisit that when this
> > bug is closed.
> 
> Sad to hear this.

As an encouragement for you to upload soon with my patch (or an improved
one skipping lots of tests), Antonio removed the blacklisting
yesterday¹, which is live as we speak.

Paul

¹
https://salsa.debian.org/ci-team/debci/commit/2d22046af8b0fc67a7ce35ce23c663737723953f



signature.asc
Description: OpenPGP digital signature

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Paul Gevers]

Hi,

On 15-03-18 17:06, Holger Levsen wrote:
> On Thu, Mar 15, 2018 at 03:44:02PM +0100, Paul Gevers wrote:
>> I believe the attached patch would be able to fix this bug and on top of
>> that make the test script a lot simpler and easier to understand. Please
>> verify that the script still does what is intended.
> 
> awesome, thanks a lot! (did you test it manually?)

I did partially. My computer stalled in the middle (not due to the
testing I believe, but due to what I fear is an Emacs bug). Mind you, I
believe the test-metapkgs actual fails. Generating logs as I write this
message, will send them when they are ready.

>> Also, I wonder if really all the current combinations of DESKTOP and
>> PROFILE need to be tested, or some smarter combination could be chosen
>> such that test time can be reduced while still having confidence that
>> most code paths are tested.
> 
> I think the latter would be sufficient indeed.
> 
>> I haven't optimized for this yet.
> 
> I'd certainly welcome another patch based on this one ;)

I'd rather have that people knowledgeable on how debian-edu code
actually works choose the right set.

Paul



signature.asc
Description: OpenPGP digital signature

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Holger Levsen]

Hi Paul,

On Thu, Mar 15, 2018 at 03:44:02PM +0100, Paul Gevers wrote:
> I believe the attached patch would be able to fix this bug and on top of
> that make the test script a lot simpler and easier to understand. Please
> verify that the script still does what is intended.

awesome, thanks a lot! (did you test it manually?)

> Please note the chmod on d/t/test-metapkgs that is required.

noted & thankfully you provided a git patch, so thats easy. :)

> Also, I wonder if really all the current combinations of DESKTOP and
> PROFILE need to be tested, or some smarter combination could be chosen
> such that test time can be reduced while still having confidence that
> most code paths are tested.

I think the latter would be sufficient indeed.

> I haven't optimized for this yet.

I'd certainly welcome another patch based on this one ;)


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Paul Gevers]

Control: tags -1 patch

On 14-03-18 10:53, Holger Levsen wrote:
>> If you want/need help to fix this bug, let me know and I'll try to have
>> a look.
>  
> we surely would appreciate help fixing this bug!

I believe the attached patch would be able to fix this bug and on top of
that make the test script a lot simpler and easier to understand. Please
verify that the script still does what is intended.

Please note the chmod on d/t/test-metapkgs that is required.

Also, I wonder if really all the current combinations of DESKTOP and
PROFILE need to be tested, or some smarter combination could be chosen
such that test time can be reduced while still having confidence that
most code paths are tested. I haven't optimized for this yet.

Paul
From c6e6f10488af80d4a76ee24ecd1bd162ae8b53b4 Mon Sep 17 00:00:00 2001
From: Paul Gevers 
Date: Thu, 15 Mar 2018 13:26:10 +0100
Subject: [PATCH] Rearrange autopkgtests such that the don't time out anymore

- run PROFILE/DESKTOP combinations separately
- get rid of debootstrap (as it isn't needed anymore)
- greatly simplify test code
---
 debian/tests/control   | 12 --
 debian/tests/test-metapkgs | 59 +++---
 2 files changed, 13 insertions(+), 58 deletions(-)
 mode change 100644 => 100755 debian/tests/test-metapkgs

diff --git a/debian/tests/control b/debian/tests/control
index e76e2a1c..d1611a5a 100644
--- a/debian/tests/control
+++ b/debian/tests/control
@@ -1,3 +1,11 @@
-Tests:  test-tasksel-desktop test-metapkgs
-Depends: debootstrap, debian-edu-install, education-common, education-tasks
+Tests: test-tasksel-desktop
+Depends: education-common
+Restrictions: needs-root allow-stderr
+
+Test-Command: PROFILE=Minimal debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
+Restrictions: needs-root allow-stderr
+
+Test-Command: PROFILE=Workstation DESKTOP=kde debian/tests/test-metapkgs
+Depends: education-common, how-can-i-help
 Restrictions: needs-root allow-stderr
diff --git a/debian/tests/test-metapkgs b/debian/tests/test-metapkgs
old mode 100644
new mode 100755
index 6a374603..6e4d4653
--- a/debian/tests/test-metapkgs
+++ b/debian/tests/test-metapkgs
@@ -1,60 +1,17 @@
 #!/bin/sh
 set -e
 
-at_exit() {
-echo "info: terminating script"
-for m in $umount ; do
-	echo info: umounting "$m"
-	umount "$m"
-done
-}
-
-trap at_exit INT TERM EXIT
-
-# Make sure temp directory setting do not leak into chroot.
-chroot_run() {
-target=$1
-shift
-TMP= TMPDIR= TEMP= TEMPDIR= chroot $target "$@"
-}
-
 chroot_test() {
 echo
 echo "Installing Debian Edu chroot with profile $PROFILE and desktop $DESKTOP"
 echo
 cd $ADTTMP
-target=test-chroot
-suite=testing
-edusuite=buster
-debootstrap $suite $target
-printf "#!/bin/sh\nexit 101\n" > $target/usr/sbin/policy-rc.d
-chmod a+rx $target/usr/sbin/policy-rc.d
-mount -t proc proc $target/proc
-umount="$target/proc"
-mount -t sysfs sysfs $target/sys
-umount="$target/proc $target/sys"
 
-# The bless script is recently part of the debian-edu-config package.
-if [ -x /usr/share/debian-edu-config/tools/debian-edu-bless ] ; then
-	cp /usr/share/debian-edu-config/tools/debian-edu-bless \
-	$target/root/.
-else
-	apt-get source debian-edu-config 2>&1
-	cp debian-edu-config-*/share/debian-edu-config/tools/debian-edu-bless \
-	$target/root/.
-fi
 PROFILE=$PROFILE DESKTOP=$DESKTOP EDUSUITE=$edusuite \
-	chroot_run $target sh -x /root/debian-edu-bless 2>&1
+sh -x /usr/share/debian-edu-config/tools/debian-edu-bless 2>&1
 
 # List packages with problems in Debian
-chroot_run $target apt-get install -y how-can-i-help
-chroot_run $target how-can-i-help --old
-
-umount $target/proc
-umount="$target/sys"
-umount $target/sys
-umount=""
-rm -rf $target
+how-can-i-help --old
 }
 
 # Use predictable language setting.
@@ -73,14 +30,4 @@ echo "info: File system status (/proc/mounts):"
 cat /proc/mounts
 echo
 
-# Install profiles in chroots, first the non-desktop ones
-for profile in Minimal Main-Server; do
-PROFILE="$profile" chroot_test
-done
-
-# Then the desktop ones
-for profile in Workstation Roaming-Workstation LTSP-Server Standalone; do
-for desktop in kde gnome lxde xfce mate ; do
-DESKTOP="$desktop" PROFILE="$profile" chroot_test
-done
-done
+chroot_test
-- 
2.16.2



signature.asc
Description: OpenPGP digital signature

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Holger Levsen]

On Tue, Mar 13, 2018 at 08:55:53PM +0100, Paul Gevers wrote:
> Any progress on this? We are on the verge of using autopkgtest results
> for unstable-to-testing migration and missing debian-edu results seems
> just like a shame.

I'm not convinced it would be such a big loss, we have other tests on
jenkins.d.n already (and should add some on openqa.d.n).

that said...

> If you want/need help to fix this bug, let me know and I'll try to have
> a look.
 
we surely would appreciate help fixing this bug!


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Paul Gevers]

On Fri, 27 Nov 2015 13:46:48 -0200 Antonio Terceiro
 wrote:
> On Fri, Nov 27, 2015 at 11:44:35AM +0100, Petter Reinholdtsen wrote:
> > [Antonio Terceiro]

>>> I am blacklisting debian-edu for now, and will revisit that when
>>> this bug is closed.>>
>> Sad to hear this.
>
> I'm not happy about it either, but right now the tests are broken 
> and the system can test hundreds or even thousands of packages during
> the time in which debian-edu would run and fail midway 100% of the
> time.
Any progress on this? We are on the verge of using autopkgtest results
for unstable-to-testing migration and missing debian-edu results seems
just like a shame.

If you want/need help to fix this bug, let me know and I'll try to have
a look.

Paul



signature.asc
Description: OpenPGP digital signature

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Antonio Terceiro]

On Fri, Nov 27, 2015 at 11:44:35AM +0100, Petter Reinholdtsen wrote:
> [Antonio Terceiro]
> > Hello,
> 
> Hi.
> 
> > The debian-edu test suite is timing out on ci.debian.net.
> 
> What is the timeout value on ci.debian.net?

The defaults for adt-run. there are different timeouts for different
parts of the process. from adt-run(1):

  There are five timeouts affected by five values of which: short: sup‐
  posedly  short operations like setting up the testbed's apt and
  checking the state (default: 100s); install: installation of packages
  including dependencies (default: 3,000s); test: test runs (default:
  10,000s); copy: copy files/directories between host  and testbed
  (default: 300s); and build: builds (default: 100,000s).

> > I noted the test suite invokes debootstrap several times, however
> > autopkgtest/DEP-8 is supposed to test the packages as installed, and I
> > don't quite understand what is the point of testing several full
> > debootstraps every time. Is there anything that really needs to be
> > tested with a full debootstrap and not with just installing the
> > relevant packages on an existing clean testbed?
> 
> The debian-edu package is a set of dynamically created metapackages that
> should be installable, and the autopkgtest scripts are verifying that
> they are indeed installable.  Not all the metapackages are
> co-installable, and installing several of them in the same testbed do
> not make sense.  This is the reason separate debootstrap environments
> are used.

running different tests against fresh testbeds is builtin in autopkgtest
since the beginning. You can just do

  Test: debian-edu-checks
  Depends: debian-edu-foo

  Test: debian-edu-checks
  Depends: debian-edu-bar

  [...]

Which will install what is in Depends:, and then -- assuming the
installation works -- call debian/tests/debian-edu-checks.

If you have to controll the installation somehow (i.e. you need to call
apt-get install explicitly with some parameters, or do something before
that), you can do like this:

  Test-Command: debian/tests/debian-edu-install-metapackage debian-edu-foo

  Test-Command: debian/tests/debian-edu-install-metapackage debian-edu-bar

Then you can concentrate your logic in the
debian-edu-install-metapackage script, which will receive the
metapackage(s) to install as command line parameters.

Each test that is declared in its own block gets a fresh, empty testbed
to start from. So no, you don't need to debootstrap every time. This has
the added feature that on a failure, just scrolling to the bottom of the
log will tell you immediately which metapackages failed and which
didn't.

> And some of the metapackages are huge (installing around 1.6 GiB of
> software, the last time I checked), so it will take a long time to
> install them.
> 
> > Also, it will help if you have separate tests (i.e. separate entries
> > in debian/tests/control) for each flavor. Since the autopkgtest
> > timeout is per-test, if you pack several very slow procedures into a
> > single test you risk getting timeouts.
> 
> This is a good idea.  I did not do it originally as I was not aware of
> such timeout problem, and wanted to keep code duplication to a minimum.

My suggestion above covers this.

> > I am blacklisting debian-edu for now, and will revisit that when this
> > bug is closed.
> 
> Sad to hear this.

I'm not happy about it either, but right now the tests are broken and
the system can test hundreds or even thousands of packages during the
time in which debian-edu would run and fail midway 100% of the time.

-- 
Antonio Terceiro 


signature.asc
Description: PGP signature

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Holger Levsen]

Hi,

On Freitag, 27. November 2015, Petter Reinholdtsen wrote:
> The debian-edu package is a set of dynamically created metapackages that
> should be installable, and the autopkgtest scripts are verifying that
> they are indeed installable.  Not all the metapackages are
> co-installable, and installing several of them in the same testbed do
> not make sense.  This is the reason separate debootstrap environments
> are used.
> 
> And some of the metapackages are huge (installing around 1.6 GiB of
> software, the last time I checked), so it will take a long time to
> install them.

you're aware of https://jenkins.debian.net/view/edu_devel/ ?


cheers,
Holger


signature.asc
Description: This is a digitally signed message part.

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Holger Levsen]

Hi,

On Freitag, 27. November 2015, Petter Reinholdtsen wrote:
> Of course.

:-)

> How come?  Do you mean we should drop the autopkgtest check
> and only use the jenkins test?  It would work, of course, but they have
> slightly different purposes

what are the different purposes?
 
> I am in doubt myself if it make sense to have both tests, but in general
> believe more quality checks are better. :)

sure! (just redudant checks are… well that. :)


cheers,
Holger




signature.asc
Description: This is a digitally signed message part.

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Petter Reinholdtsen]

[Holger Levsen]
> what are the different purposes?

The debian/tests/ scripts check installabability and desktop selection
using different mechanism from the one used by jenkins.  I would say the
purpose of the autopkgtest scripts is to check that the metapackages
work as they should (ie have the corret tasksel hooks in place), while
the jenkins scripts is to check if the dependencies of the metapackages
are installable.  They tend to fail both at the same time, but they
might also fail independenty, depending on the error situation.

I still do not quite understand why did you bring in the jenkins test
into this discussion?  Can you explain what you had in mind?

-- 
Happy hacking
Petter Reinholdtsen

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Petter Reinholdtsen]

[Holger Levsen]
> you're aware of https://jenkins.debian.net/view/edu_devel/ ?

Of course.  How come?  Do you mean we should drop the autopkgtest check
and only use the jenkins test?  It would work, of course, but they have
slightly different purposes

I am in doubt myself if it make sense to have both tests, but in general
believe more quality checks are better. :)

-- 
Happy hacking
Petter Reinholdtsen

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Holger Levsen]

Hi,

On Freitag, 27. November 2015, Petter Reinholdtsen wrote:
> The debian/tests/ scripts check installabability and desktop selection
> using different mechanism from the one used by jenkins.  I would say the
> purpose of the autopkgtest scripts is to check that the metapackages
> work as they should (ie have the corret tasksel hooks in place), while
> the jenkins scripts is to check if the dependencies of the metapackages
> are installable.  They tend to fail both at the same time, but they
> might also fail independenty, depending on the error situation.

so b includes testing a :)
 
> I still do not quite understand why did you bring in the jenkins test
> into this discussion?  Can you explain what you had in mind?

exactly what I asked, whether you were aware of them, as they seem to do quite 
the same and it would be a pity if one of them were rather useless, if nobody 
were watching.

I'm glad the jenkins tests are still useful (and happy to add more).


cheers,
Holger




signature.asc
Description: This is a digitally signed message part.

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Antonio Terceiro]

Source: debian-edu
Version: 1.812
Severity: normal
User: autopkgtest-de...@lists.alioth.debian.org
Usertags: autopkgtest

Hello,

The debian-edu test suite is timing out on ci.debian.net.

I noted the test suite invokes debootstrap several times, however
autopkgtest/DEP-8 is supposed to test the packages as installed, and I
don't quite understand what is the point of testing several full
debootstraps every time. Is there anything that really needs to be
tested with a full debootstrap and not with just installing the relevant
packages on an existing clean testbed?

Also, it will help if you have separate tests (i.e. separate entries in
debian/tests/control) for each flavor. Since the autopkgtest timeout is
per-test, if you pack several very slow procedures into a single test
you risk getting timeouts.

I am blacklisting debian-edu for now, and will revisit that when this
bug is closed.

-- System Information:
Debian Release: stretch/sid
  APT prefers buildd-unstable
  APT policy: (500, 'buildd-unstable'), (500, 'unstable'), (500, 'testing'), 
(1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.2.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=pt_BR.UTF-8, LC_CTYPE=pt_BR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

-- 
Antonio Terceiro 


signature.asc
Description: PGP signature

Bug#806420: debian-edu: test suite times out on ci.debian.net

[Petter Reinholdtsen]

[Antonio Terceiro]
> Hello,

Hi.

> The debian-edu test suite is timing out on ci.debian.net.

What is the timeout value on ci.debian.net?

> I noted the test suite invokes debootstrap several times, however
> autopkgtest/DEP-8 is supposed to test the packages as installed, and I
> don't quite understand what is the point of testing several full
> debootstraps every time. Is there anything that really needs to be
> tested with a full debootstrap and not with just installing the
> relevant packages on an existing clean testbed?

The debian-edu package is a set of dynamically created metapackages that
should be installable, and the autopkgtest scripts are verifying that
they are indeed installable.  Not all the metapackages are
co-installable, and installing several of them in the same testbed do
not make sense.  This is the reason separate debootstrap environments
are used.

And some of the metapackages are huge (installing around 1.6 GiB of
software, the last time I checked), so it will take a long time to
install them.

> Also, it will help if you have separate tests (i.e. separate entries
> in debian/tests/control) for each flavor. Since the autopkgtest
> timeout is per-test, if you pack several very slow procedures into a
> single test you risk getting timeouts.

This is a good idea.  I did not do it originally as I was not aware of
such timeout problem, and wanted to keep code duplication to a minimum.

> I am blacklisting debian-edu for now, and will revisit that when this
> bug is closed.

Sad to hear this.

-- 
Happy hacking
Petter Reinholdtsen