Hello, Am Sonntag, 25. Juni 2017, 11:18:39 CEST schrieb intrigeri: > autopkgtests are… > > "as-installed" tests of packages, i.e. the testing of packages in > a context as close as possible to a Debian system where the packages > subject to testing are properly installed.
For a nice definition of "as close to possible to a real system", have a look at openQA [1]. It does everything from installing or upgrading the distribution up to things like opening websites in firefox, clicking buttons, comparing screenshots with the actual screen content etc. BTW: Fedora also started to use openQA after some SUSE people started testing Fedora "just for fun" and found quite some bugs ;-) If you are crazy enough to want this, openQA can even test Windows ;-) > Yesterday I've added some autopkgtests to the apparmor Debian package: > > > https://alioth.debian.org/scm/loggerhead/collab-maint/apparmor/revisi > on/1630 > > For now I'm running tests (with USE_SYSTEM=1) in these directories: > binutils, parser. You can/should also run the utils tests ;-) > Distro maintainers, are you doing similar things? > > I'm particularly interested in: > > * examples of how to run more tests against the installed version, > ideally outside of a source tree that was built already; Not exactly what you asked for, but it might still be interesting: For the openSUSE package, I run the checks in-tree (without USE_SYSTEM). %check %if %{with python3} export PYTHON=/usr/bin/python3 export PYTHON_VERSIONS=python3 %endif make check -C libraries/libapparmor make check -C parser make check -C binutils # profiles make check fails for the utils (libapparmor PYTHONPATH # issues), therefore only do parser-based checks # also, check-parser breaks if using 'make -C' (but works if cd'ing into # the directory) (cd profiles && make check-parser) make check -C utils BTW: If someone has an idea about the strange make -C issue for the profiles directory - patches welcome ;-) Same for the PYTHONPATH issue. Maybe packaging all the tests into an apparmor-tests package, and then building a dummy package that runs the tests (without having the source tree available) would be an option. However, I'm not sure if the gain is worth this effort. I also asked to get some tests included in openQA (which would mean to have them run each time a new Tumbleweed snapshot gets released - but unfortunately the openQA team is too busy, so I'm still on the waiting list to get this done :-( The tests I want in openQA are more user-centric: function errorexit() { echo "*** $1 ***" >&2 exit 1 } export LC_ALL=C # make sure all messages are in english, because we grep for some of them aa-status --enabled || errorexit "AppArmor not enabled" test `aa-status --enforced` -gt 0 || errorexit "no profiles loaded in enforce mode" # make sure all running processes have their profile applied aa-status | grep '^0 processes are unconfined but have a profile defined.' || { aa-status; errorexit "unconfined processes with a profile defined"; } # test ping profile ping -c1 127.0.0.1 || errorexit "ping IPv4 failure" # Leap 42.x has an older IPv4-only ping (and a separate ping6 command, which we don't test here) # ping in Tumbleweed can ping IPv6 adresses test $(rpm --eval '%suse_version') -gt 1320 && { ping -c1 '::1' || errorexit "ping IPv6 failure"; } # test if mktemp works tempfile=$(mktemp) || errorexit "mktemp failed (without AppArmor restrictions)" rm -f "$tempfile" # provoke a failure by using mktemp with the klogd profile (which isn't allowed to write in /tmp) mktempmsg=$(aa-exec -p klogd mktemp 2>&1) test $? == 1 || errorexit "mktemp did not fail with the klogd profile" echo "$mktempmsg" | grep 'Permission denied' || { echo "$mktempmsg"; errorexit "mktemp didn't fail with 'Permission denied'"; } # make sure aa-complain works and really sets the profile into complain mode aa-complain /etc/apparmor.d/sbin.klogd || errorexit "aa-complain failed" # this will also allow mktemp to create a tempfile tempfile=$(aa-exec -p klogd mktemp 2>&1) || errorexit "mktemp with complain mode klogd profile failed" rm -f "$tempfile" # switch profile back to enforce mode aa-enforce /etc/apparmor.d/sbin.klogd || errorexit "aa-enforce failed" systemctl status apparmor.service || errorexit "systemctl status failed" systemctl is-enabled apparmor.service || errorexit "systemctl is-enabled failed" echo echo "AppArmor tests successful :-)" > * whether Ubuntu has other autopkgtests, and where they are defined. Ubuntu runs lots of AppArmor tests as part of https://launchpad.net/qa-regression-testing/ I played with these tests (only the AppArmor-related ones) on openSUSE a while ago. IIRC I found two generic problems that prevented them from "simply running": - Ubuntu creates a group for each user, while openSUSE doesn't do this and has a "users" group instead. This of course results in different behaviour for the pam_apparmor tests. - tests for dbus etc. won't work - this should be fixed as soon as all the kernel patches got upstreamed. Regards, Christian Boltz [1] https://openqa.opensuse.org/ and http://open.qa -- a computer without an Internet connection is essentially a very expensive DVD player [http://www.randsinrepose.com/archives/2006/07/10/a_nerd_in_a_cave.html]
signature.asc
Description: This is a digitally signed message part.
-- AppArmor mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/apparmor
