On Thu, 2013-04-25 at 16:30 +0200, Bastien Nocera wrote: > What are the requirements in terms of what tests are allowed to do? I'm > guessing they shouldn't touch the running environment,
That's a good question. The gnome-ostree testing infrastructure uses qemu with a copy-on-write qcow2 disk snapshot for all tests; they're run as an autologged in user with an active desktop session. It'd absolutely be possible to have tests which mutate the environment, or even run as root and possibly trash the system. But the primary thing I'm focusing on now is transitioning most of GNOME's set of tests which tend to have these characteristics: * Run as non root * Assume the presence of a logged in session (or sometimes autocreate one with Xvfb/dbus-launch) * Are nondestructive Almost all of the GLib, gjs, and pygobject test cases are of that form; by having them be installed, builders and integrators can fully automate running them however often they choose. Though you've got me thinking: maybe instead of the /usr/bin/foo-installed-test naming scheme, we should have /usr/share/installed-tests/foo.desktop. Desktop files by their nature already: 1) Are what we use to launch binaries 2) Have metadata If other people like that idea, I'll look at reworking the testing code to use .desktop files. > create files in > the user's home directory, etc. Are we allowed to access the network for > example? I don't think we should write network-dependent tests at this time; they'd be a huge source of unreliability. Now it'd obviously be very cool to have e.g. a static copy of a wikipedia page hosted in GNOME infrastructure, and have a test which automates running Epiphany against that page. But let's learn to walk before we run =) > Do we have (makefile) infrastructure to allow GTests to run both > uninstalled (during make *check) and installed? Not at this time; that'd be nice, but I think the jhbuild model mostly obviates the need for uninstalled tests, because jhbuild by nature is a sandboxed environment independent from the underlying system. cd ~/src/glib jhbuild make # note: runs make install ~/build/bin/glib-proxy-installed-test So if you broke glib, you still have the underlying system to fall back on. Now for things like NetworkManager that jhbuild is incapable of handling, I can certainly imagine how uninstalled tests would be very convenient for developers. And it would likely make sense for some of those tests to support running them both installed and uninstalled. But the point of the gnome-ostree model is precisely to test those types of things that have historically had very bad testing coverage because jhbuild is incapable of handling them, and "sudo make install" and hacking up distribution packages also sucks. _______________________________________________ desktop-devel-list mailing list [email protected] https://mail.gnome.org/mailman/listinfo/desktop-devel-list
