Re: New GnomeGoal proposal: InstalledTests
So, two weeks later. A quick status update on prototyping this: First, I converted gjs to have an installed-tests subdirectory: https://git.gnome.org/browse/gjs/commit/?id=54538f3da91b25ed20c9deaeda00d57fb252227c That worked, however I got some pushback: https://bugzilla.gnome.org/show_bug.cgi?id=698935 Meanwhile, I tried doing this for both glib: https://bugzilla.gnome.org/show_bug.cgi?id=699079 and GTK+: https://bugzilla.gnome.org/show_bug.cgi?id=699601 Of those, the glib patches are unreviewed as of yet (but work in local testing). Much more exciting is that I now have the GTK+ reftests under automation; they run after every build, and I'm taking screenshots along the way: http://build.gnome.org/ostree/buildmaster/tasks/integrationtest/saved-20130510.1/work-gnome-ostree-gnome-ostree_buildmaster_x86_64-runtime-disk/screenshot-7.png (I manually saved that particular task run, otherwise it'd eventually be rotated out by later integration test runs) I also converted the ostree tests to be installed: https://git.gnome.org/browse/ostree/commit/?id=e9a181c67d7d9bccc4aafd172b14996ae4dc5d43 So, is this useful and worth the additional maintenance as a different option from make check? I strongly believe so. For the core modules like glib/gtk+, it definitely felt conclusive from this thread that we should keep make check working. Sometimes that was quite un-obvious and tricky, and so for my module ostree I decided to just burn the bridges and have the tests only work installed. Next steps: * Add --enable-installed-tests to jhbuild * Teach Debian's Autopkgtest how to run these * Improve gnome-desktop-testing's runner binary gnome-desktop-testing-runner * More tests. More! The specification is still open to revision of course too. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Fri, 2013-04-26 at 22:47 +0900, Tristan Van Berkom wrote: On Fri, Apr 26, 2013 at 9:46 PM, Matthias Clasen matthias.cla...@gmail.com wrote: On Thu, Apr 25, 2013 at 10:45 AM, Colin Walters walt...@verbum.org wrote: Do we have (makefile) infrastructure to allow GTests to run both uninstalled (during make *check) and installed? Not at this time; that'd be nice, but I think the jhbuild model mostly obviates the need for uninstalled tests, because jhbuild by nature is a sandboxed environment independent from the underlying system. You are not going to get me to buy eagerly into a new installed tests scheme for glib if it means that I have to give up make check. I just wanted to jump in and mention that, I'd really really like to see better all around relocatability of modules. Ideally, I'd like to see the following things: o Ability to run installed tests, encapsulated in the jhbuild prefix o Ability to use the same test cases that I use in-tree as installed tests, hopefully by simply installing my in-tree tests somewhere (perhaps they would run without some of the custom env vars that I would use in-tree, but still re-use all the same test suites when installed) o Overall relocated D-Bus and settings environment This last one is a really big deal, there are some really annoying things right now, for instance I can in *no way* test GTK+ apps in a jhbuild shell and use the theme installed in the jhbuild prefix. It seems natural that when typing 'jhbuild shell', any processes that run should be accessing the settings persisted somewhere in /opt/devel... instead I would have to slave my whole system just to see what an app looks like with the theme installed in /opt/devel. This problem applies to a whole classes of Gnome development. I just lost of a lot of time yesterday trying to set up a development environment in a VM so I would have a full, graphical session (because apparently parts of the Shell are what normally start up some EDS components, so 'jhbuild shell' is insufficient). I still don't have a working solution here and I've lost track of the times I've tried to resolve this with existing components without success. Any long-term solution needs to be based within a standard VM setup that most developers (core and third-party) all use (for dog-fooding purposes). Many developers can get away with just using jhbuild now, but it's too different from a regular session for other developers. I think there was general agreement on this idea in the Gnome Developer Experience Hackfest in February. We need an SDK that is the official and de facto standard for first- and third-party Gnome development so we can all benefit from solutions to our common development problems. GTestDBus has brought us at least part of the way there when it comes to D-Bus, but if we run installed tests we'll want to disable the relocation of services done by GTestDBus (so that some apps and services can interact with other services installed into the build prefix, instead of running in complete isolation which is more appropriate for 'make check'). Perhaps this can be done by hardwiring some features directly into GTestDBus, of course in this case we still would want to run a separate session bus for the jhbuild environment, so that the whole relocated installed test suite does not interfere with a real active session. When I'm developing, I honestly only care how my new code works when it's installed. I use build-tree tests to save the time of installing, starting a new session for the development user (if necessary), etc. The length of the development cycle between each edit and verifying it is critical. The difference between a 3s and 1m loop is huge in my productivity. But if a 1-minute loop meant not fighting issues related to build-tree vs. installed, full graphical session vs. jhbuild session, etc. issues, I'd very gladly take it. And 1 minute is just an arbitrary number. I'm sure we could minimize the lag added by installation, session initiation, etc. I also realize this is a lot of work, but frankly better relocatability is something that benefits our workflow (I would be able to actually run Evolution Mail in jhbuild without interfering with my mail account, I would be able to see what GTK+ apps look like to those select few who are actually running the bleeding edge of GNOME on a device)... i.e. it would not only beneficial for installed tests, but it would help with installed tests too. Any real solution is going to take a lot of work, which is probably why it hasn't happened yet. But we need something along the lines of OSTree + VM + a simple, standard method to create fast push-button compile-install-verify cycles
Re: New GnomeGoal proposal: InstalledTests
On Tue, 2013-04-30 at 10:27 -0700, Travis Reitter wrote: On Fri, 2013-04-26 at 22:47 +0900, Tristan Van Berkom wrote: On Fri, Apr 26, 2013 at 9:46 PM, Matthias Clasen matthias.cla...@gmail.com wrote: On Thu, Apr 25, 2013 at 10:45 AM, Colin Walters walt...@verbum.org wrote: Do we have (makefile) infrastructure to allow GTests to run both uninstalled (during make *check) and installed? Not at this time; that'd be nice, but I think the jhbuild model mostly obviates the need for uninstalled tests, because jhbuild by nature is a sandboxed environment independent from the underlying system. You are not going to get me to buy eagerly into a new installed tests scheme for glib if it means that I have to give up make check. I just wanted to jump in and mention that, I'd really really like to see better all around relocatability of modules. Ideally, I'd like to see the following things: o Ability to run installed tests, encapsulated in the jhbuild prefix o Ability to use the same test cases that I use in-tree as installed tests, hopefully by simply installing my in-tree tests somewhere (perhaps they would run without some of the custom env vars that I would use in-tree, but still re-use all the same test suites when installed) o Overall relocated D-Bus and settings environment This last one is a really big deal, there are some really annoying things right now, for instance I can in *no way* test GTK+ apps in a jhbuild shell and use the theme installed in the jhbuild prefix. It seems natural that when typing 'jhbuild shell', any processes that run should be accessing the settings persisted somewhere in /opt/devel... instead I would have to slave my whole system just to see what an app looks like with the theme installed in /opt/devel. This problem applies to a whole classes of Gnome development. I just lost of a lot of time yesterday trying to set up a development environment in a VM so I would have a full, graphical session (because apparently parts of the Shell are what normally start up some EDS components, so 'jhbuild shell' is insufficient). I still don't have a working solution here and I've lost track of the times I've tried to resolve this with existing components without success. Any long-term solution needs to be based within a standard VM setup that most developers (core and third-party) all use (for dog-fooding purposes). Many developers can get away with just using jhbuild now, but it's too different from a regular session for other developers. I think there was general agreement on this idea in the Gnome Developer Experience Hackfest in February. We need an SDK that is the official and de facto standard for first- and third-party Gnome development so we can all benefit from solutions to our common development problems. GTestDBus has brought us at least part of the way there when it comes to D-Bus, but if we run installed tests we'll want to disable the relocation of services done by GTestDBus (so that some apps and services can interact with other services installed into the build prefix, instead of running in complete isolation which is more appropriate for 'make check'). Perhaps this can be done by hardwiring some features directly into GTestDBus, of course in this case we still would want to run a separate session bus for the jhbuild environment, so that the whole relocated installed test suite does not interfere with a real active session. When I'm developing, I honestly only care how my new code works when it's installed. I use build-tree tests to save the time of installing, starting a new session for the development user (if necessary), etc. The length of the development cycle between each edit and verifying it is critical. The difference between a 3s and 1m loop is huge in my productivity. But if a 1-minute loop meant not fighting issues related to build-tree vs. installed, full graphical session vs. jhbuild session, etc. issues, I'd very gladly take it. And 1 minute is just an arbitrary number. I'm sure we could minimize the lag added by installation, session initiation, etc. In the nearer-term, if we could optimize 'make check' for quick sanity checks (particularly if we could work out minimum dependencies automatically) and run installed tests for a more comprehensive test, I would probably move a fair number of the Folks tests to be installed. That could speed up my development loop quite a bit without too much work and would buy more time to speed up the comprehensive path installed test loop. (They may be fundamentally slower, especially if we emphasized a quick vs. comprehensive split, but it'd be
Re: New GnomeGoal proposal: InstalledTests
On Thu, 2013-04-25 at 10:21 -0400, Colin Walters wrote: See: https://live.gnome.org/GnomeGoals/InstalledTests I'm in the process of transitioning to that page from https://docs.google.com/document/d/1DXgGEKTbC4ed1DFW3Mu-48TpHtq-xdfhDtwoUOWWcHg/edit Comments appreciated; I have prototype branches of gnome-ostree, gnome-desktop-testing, and gjs that implement this, but it's all subject to change. If the test are going to be installed by default it might be reasonable to have a common option (similar to gtk doc) to disable them (if they are not going to be enabled by default - option to enable them) in case user does not need them. Say --enable-install-tests=yes/no. Best regards ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Mon, 2013-04-29 at 08:07 +0100, Maciej Piechotka wrote: If the test are going to be installed by default it might be reasonable to have a common option (similar to gtk doc) to disable them (if they are not going to be enabled by default - option to enable them) in case user does not need them. Say --enable-install-tests=yes/no. Yes; see the section Source code in the current spec. It calls for them to be disabled by default, and a configure option named --enable-installed-tests to exist. That's just one of the options though. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
Hello all, Colin Walters [2013-04-28 19:26 -0400]: But note that if we're talking about black box testing which is what most of GNOME's tests are, the II case is effectively a superset of all of the other cases; it's the most flexible and powerful. Agreed, but at the cost of also being the most expensive to use. You're talking about this II case as something daily. But I've optimized the gnome-ostree build system now to the point where the build server consistently gets to running installed tests in qemu 3-4 minutes (sometimes less) after a git commit. This is running in and testing the real tree that users will download; it's not a weird partially assembled pseudo-environment like jhbuild or a mock/pbuilder root. This isn't hypothetical - while we were at the GTK+ hackfest this week the gnome-ostree system detected that https://git.gnome.org/browse/glib/commit/?id=ddb0ce14215cd62c7a2497d6cf9f2ea63c40ebb5 broke gnome-shell in minutes. And this number is only going to go down; with various optimizations, I'm pretty sure I could get a case like a git commit to gnome-shell that only modifies a .js file booting within 30 seconds of the push notification of the commit. For example, by reusing a cached copy of the previous build, so it's just a matter of make make install. At that speed, a lot of the other test forms become superfluous. That turnaround speed is indeed great for doing CI. I do agree that this is the right thing to do for finding regressions after they happened (like above GTK commit). But that's not the normal TDD mode of developing a new feature or a bug fix -- you want to write a test, then fix the bug/implement the feature with ideally zero overhead for installing, building packages/snapshots, and booting VMs, and most importantly you don't want to commit and push until everything is succeeding again. At the moment in GNOME we have a lot of BB, and I'm pushing hard for II. Per smcv's comment, I believe projects like Debian could effectively reuse II tests in autopkgtest by simply ignoring the source tree aspect and running the installed binaries. Yes, absolutely. The II bit will make things like autopkgtest dramatically easier, faster, and more reliable. I fully agree, and I don't think there is much reason to drop uninstalled tests in favor of installed ones; In most cases it should be easy to support both. That's the model how development (should) work, after all: You write a test for a new feature, then implement it, and run make check (or sudo make check) until it works; it would be a step backwards to require installing the tests (and clobbering your system) first. But what if you had a build system that made it really easy and reasonably fast to go from list of git repositories to bootable VM? I do, and it's changed my whole perspective on development and testing =) For things like NM or systemd, this is definitively a good thing. Also, for testing a complete system after making a fundamental change in e. g. glib. But I consider it too much overhead for the 20 build/test/fix iterations for doing a bug fix or new feature. Anyways, as far as an executive summary here: Module maintainers can/should keep supporting BB at their discretion, but are you OK with the proposed II model? Of course; sorry if that didn't come across, I really like the proposal. I just wanted to point out that it is not really desirable nor necessary to drop the make check (BB) case for this. make check and calling the tests in a VM should ideally call the very same test suite, just with/without LD_LIBRARY_PATH and friends set. Thanks! Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Sat, 2013-04-27 at 15:40 +0200, Martin Pitt wrote: (II) installed tests on installed system: regular verification of daily image builds, self-check of installed systems and ostree snapshots But note that if we're talking about black box testing which is what most of GNOME's tests are, the II case is effectively a superset of all of the other cases; it's the most flexible and powerful. It also is the one that is easiest to match closely with what normal users actually get. You're talking about this II case as something daily. But I've optimized the gnome-ostree build system now to the point where the build server consistently gets to running installed tests in qemu 3-4 minutes (sometimes less) after a git commit. This is running in and testing the real tree that users will download; it's not a weird partially assembled pseudo-environment like jhbuild or a mock/pbuilder root. This isn't hypothetical - while we were at the GTK+ hackfest this week the gnome-ostree system detected that https://git.gnome.org/browse/glib/commit/?id=ddb0ce14215cd62c7a2497d6cf9f2ea63c40ebb5 broke gnome-shell in minutes. And this number is only going to go down; with various optimizations, I'm pretty sure I could get a case like a git commit to gnome-shell that only modifies a .js file booting within 30 seconds of the push notification of the commit. For example, by reusing a cached copy of the previous build, so it's just a matter of make make install. At that speed, a lot of the other test forms become superfluous. They were defensive measures necessary because package-based systems suck at continuous integration. So it would be good if we can have all four and provide standard ways of running them. At the moment in GNOME we have a lot of BB, and I'm pushing hard for II. Per smcv's comment, I believe projects like Debian could effectively reuse II tests in autopkgtest by simply ignoring the source tree aspect and running the installed binaries. IB can be subsumed by II if the build system allows just installing the libraries, or just installing the tests. Something like: $ cd ~/src/glib $ jhbuild make --enable-installed-tests # builds and installs glib *and* tests $ hack hack hack $ jhbuild make --disable-installed-tests Now you have a newer libglib.so in your build root, on which you can run the previous binary tests. I fully agree, and I don't think there is much reason to drop uninstalled tests in favor of installed ones; In most cases it should be easy to support both. That's the model how development (should) work, after all: You write a test for a new feature, then implement it, and run make check (or sudo make check) until it works; it would be a step backwards to require installing the tests (and clobbering your system) first. But what if you had a build system that made it really easy and reasonably fast to go from list of git repositories to bootable VM? I do, and it's changed my whole perspective on development and testing =) Anyways, as far as an executive summary here: Module maintainers can/should keep supporting BB at their discretion, but are you OK with the proposed II model? ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
Simon McVittie [2013-04-25 16:41 +0100]: I think the default should be that tests are non-destructive, run as non-root, do not require a session, do not require networking and so on; if any of these are not true, they should be declared via something analogous to autopkgtest's Restrictions. Right, and each of these limitations makes it harder (and thus less likely) for developers to actually run them regularly. Running tests as root is not much of a drag, but jumping from sudo make check to have to build and boot a VM and run tests there is a major degradation of turnaround time. Fortunately there are nice and proven ways to avoid touching the production system's bits, like - GSETTINGS_BACKEND=memory - Set $HOME, $XDG_CONFIG_HOME, $XDG_DATA_HOME, $XDG_RUNTIME_DIR to temporary locations (especially $HOME now works with glib 2.36) (example: gvfs) - Run stuff on a temporary private session and/or system D-BUS (example: upower, gvfs) - If root tests need to change some files in /etc, /var/lib/foo/ etc., run them through 'unshare -m' and put tmpfs mounts over these paths (example: gvfs) - If the program depends on a particular state/behaviour of things like logind or polkit, use python-dbusmock to mock them, instead of actually creating active/inactive sessions or actually suspending the machine (example: gnome-settings-daemon, udisks) - If you need to talk to a network server, start one locally with a local apache/ssh/ftp/etc. server (example: gvfs) - If you need to format/create/remove/hotplug block devices, use scsi_debug instead of trashing the actual hard disk (example: udisks) - If you need to test network device handling, use mac80211_hwsim/hostapd/dnsmasq instead of trashing the actual system's network (example: upcoming NetworkManager tests; they are not upstream yet, you can look at them at [1] for now. I found that in most cases you can get away with not having to run tests in a VM, and still retain the possibility of doing a lot of those intrusive kinds of test bed setup. Martin [1] http://bazaar.launchpad.net/~network-manager/network-manager/ubuntu/files/head:/debian/tests/ -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
Colin Walters [2013-04-25 10:45 -0400]: Now for things like NetworkManager that jhbuild is incapable of handling, I can certainly imagine how uninstalled tests would be very convenient for developers. I fully agree, and I don't think there is much reason to drop uninstalled tests in favor of installed ones; In most cases it should be easy to support both. That's the model how development (should) work, after all: You write a test for a new feature, then implement it, and run make check (or sudo make check) until it works; it would be a step backwards to require installing the tests (and clobbering your system) first. And it would likely make sense for some of those tests to support running them both installed and uninstalled. I think this installed vs. uninstalled distinction should apply to both the tests, as well as the test subject. automake has this standard target installcheck which should mean test the installed binaries, not the ones from the build tree. All four combinations are useful: (BB) build tree tests on build tree binaries: normal mode of development (BI) build tree tests on installed binaries: that's how e. g. Debian's autopkgtest works, and verifies that make install and the packaging actually works (e. g. you put stuff in the right place and didn't forget to install files); this is also useful to check how your potentially older installed version of the software holds up against your refined tests in trunk. (IB) installed tests on build tree binaries: verify backwards compatibility, i. e. whether you broke API/behaviour (II) installed tests on installed system: regular verification of daily image builds, self-check of installed systems and ostree snapshots So it would be good if we can have all four and provide standard ways of running them. Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Thu, Apr 25, 2013 at 10:45 AM, Colin Walters walt...@verbum.org wrote: Do we have (makefile) infrastructure to allow GTests to run both uninstalled (during make *check) and installed? Not at this time; that'd be nice, but I think the jhbuild model mostly obviates the need for uninstalled tests, because jhbuild by nature is a sandboxed environment independent from the underlying system. You are not going to get me to buy eagerly into a new installed tests scheme for glib if it means that I have to give up make check. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Fri, Apr 26, 2013 at 9:46 PM, Matthias Clasen matthias.cla...@gmail.comwrote: On Thu, Apr 25, 2013 at 10:45 AM, Colin Walters walt...@verbum.org wrote: Do we have (makefile) infrastructure to allow GTests to run both uninstalled (during make *check) and installed? Not at this time; that'd be nice, but I think the jhbuild model mostly obviates the need for uninstalled tests, because jhbuild by nature is a sandboxed environment independent from the underlying system. You are not going to get me to buy eagerly into a new installed tests scheme for glib if it means that I have to give up make check. I just wanted to jump in and mention that, I'd really really like to see better all around relocatability of modules. Ideally, I'd like to see the following things: o Ability to run installed tests, encapsulated in the jhbuild prefix o Ability to use the same test cases that I use in-tree as installed tests, hopefully by simply installing my in-tree tests somewhere (perhaps they would run without some of the custom env vars that I would use in-tree, but still re-use all the same test suites when installed) o Overall relocated D-Bus and settings environment This last one is a really big deal, there are some really annoying things right now, for instance I can in *no way* test GTK+ apps in a jhbuild shell and use the theme installed in the jhbuild prefix. It seems natural that when typing 'jhbuild shell', any processes that run should be accessing the settings persisted somewhere in /opt/devel... instead I would have to slave my whole system just to see what an app looks like with the theme installed in /opt/devel. GTestDBus has brought us at least part of the way there when it comes to D-Bus, but if we run installed tests we'll want to disable the relocation of services done by GTestDBus (so that some apps and services can interact with other services installed into the build prefix, instead of running in complete isolation which is more appropriate for 'make check'). Perhaps this can be done by hardwiring some features directly into GTestDBus, of course in this case we still would want to run a separate session bus for the jhbuild environment, so that the whole relocated installed test suite does not interfere with a real active session. Some few tests, I realize will always depend on how the system is setup, perhaps require a specific compositor, or even wayland, but surely 90% or more of the stack is software that is not system dependant. It would be interesting to have a separate class of installed tests which *absolutely* cannot run in a relocated environment, but those would really be a corner minimum case. I also realize this is a lot of work, but frankly better relocatability is something that benefits our workflow (I would be able to actually run Evolution Mail in jhbuild without interfering with my mail account, I would be able to see what GTK+ apps look like to those select few who are actually running the bleeding edge of GNOME on a device)... i.e. it would not only beneficial for installed tests, but it would help with installed tests too. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Thu, Apr 25, 2013 at 10:21:47AM -0400, Colin Walters wrote: https://live.gnome.org/GnomeGoals/InstalledTests I'm not entirely sure what sort of tests your framework is designed to contain... Have you seen Brian Nitz's work on GNOME accessibility tests that can be automatically run from a tinderbox? documentation: http://bazaar.launchpad.net/~brian-nitz/a11ytesting/trunk/view/head:/a11y/doc/AegistestFrameworkIntro1.0.4.pdf code: https://code.launchpad.net/~brian-nitz/a11ytesting/trunk Cheers, Patrick ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Fri, 2013-04-26 at 08:46 -0400, Matthias Clasen wrote: You are not going to get me to buy eagerly into a new installed tests scheme for glib if it means that I have to give up make check. Well, would you be OK with: $ jhbuild make $ gnome-desktop-testing-runner glib ? Note though that not all the GLib tests are of the installed form; for example, glib/tests/1bit-mutex.c actually includes some of the GLib source code again and rebuilds it in a different configuration. That sort of thing is what I'd call a white box unit test. It knows about (and reuses) the implementation details. But most of GLib's test are black box, and those are the sort that make sense to install, because that's the best way to mimic a black box scenario. But anyways I don't plan to do anything with GLib soon until I've made Giovanni happy again in: https://bugzilla.gnome.org/show_bug.cgi?id=698935 You have to understand the win here though - if we install the tests, they can be easily fully automated. Since I've switched the gjs test suite to be installed, it's already run over 28 times just so far today (UTC), without any human intervention. Also, debugging installed tests is *so* much nicer than make check - you don't have to fight through layers of generated Makefile and libtool wrapping your actual binary if you want to run gdb/valgrind on it. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On 04/26/2013 10:12 AM, Colin Walters wrote: On Fri, 2013-04-26 at 08:46 -0400, Matthias Clasen wrote: You are not going to get me to buy eagerly into a new installed tests scheme for glib if it means that I have to give up make check. Well, would you be OK with: $ jhbuild make $ gnome-desktop-testing-runner glib I want make distcheck to still run all of my tests, to guarantee that everything works correctly when built from a tarball, not just when built from git. Note though that not all the GLib tests are of the installed form; for example, glib/tests/1bit-mutex.c actually includes some of the GLib source code again and rebuilds it in a different configuration. It forces the code into the you don't have futexes fallback mode, so that we can test that that codepath works, even on machines that do have futexes. We absolutely want to include that as part of the installed tests too. You have to understand the win here though I understand the win of installed tests, but that doesn't mean that make check tests don't have their own set of wins. I want both. Also, debugging installed tests is *so* much nicer than make check - you don't have to fight through layers of generated Makefile and libtool wrapping your actual binary if you want to run gdb/valgrind on it. danw@laptop:~ cat ~/bin/gdb #!/bin/sh if [ -n $1 ] [ -x $1 ]; then if grep -q 'Generated by.*libtool' $1; then exec libtool --mode=execute /usr/bin/gdb --args $@ else exec /usr/bin/gdb --args $@ fi else exec /usr/bin/gdb $@ fi danw@laptop:~ :-) -- Dan ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On 26/04/13 15:12, Colin Walters wrote: On Fri, 2013-04-26 at 08:46 -0400, Matthias Clasen wrote: You are not going to get me to buy eagerly into a new installed tests scheme for glib if it means that I have to give up make check. I think it's worth having both, with the majority of tests working either way (see: libdbus, for which I'd be happy to review additions to the installed tests infrastructure, particularly if you review my pending patches in exchange). make check is much more convenient when you're just hacking on one particular project and not using the full jhbuild/ostree/whatever environment, and make distcheck in particular is a great sanity-check for is this release candidate viable?. I realise some parts of GNOME are sufficiently entangled in terms of X needs the latest snapshot of Y that you essentially can't build the development branch without at least a mini-distribution like jhbuild, but other parts - particularly low-ish-level libraries like GLib, libsoup, telepathy-glib - are perfectly reasonable to develop from a plain git checkout. Since I've switched the gjs test suite to be installed, it's already run over 28 times just so far today (UTC), without any human intervention. I don't see any reason why this couldn't be equally true for make check: the Ubuntu Jenkins setup seems to be running make check for everything, in jhbuild. (Admittedly, it seems it mostly fails; but that's not the test framework's fault.) S ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Fri, 2013-04-26 at 15:47 +0100, Simon McVittie wrote: I don't see any reason why this couldn't be equally true for make check: Because make check requires rebuilding the module just to run the tests. I'm running the gjs test suite at the moment on the complete resulting installed tree (that users will download) for *any* source change. Obviously it's not strictly *necessary* to run the gjs test suite when there's commits to xorg-xserver or gdm. But it also doesn't hurt. And it's really important to have the gjs tests run automatically (and quickly) when glib changes for example. I have plenty of CPU to go around at the moment even on this one test machine; it spends probably 60% its time totally idle, waiting for git commits. Yes, if I was doing complete reverse dependency rebuilds, that could as a consequence trigger a rerun of all tests, but full revdep rebuilds would massively slow down the feedback cycle. The point of this system is to detect bad commits *quickly* (minutes, not hours or days), even at some cost to correctness. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Fri, 2013-04-26 at 10:32 -0400, Dan Winship wrote: On 04/26/2013 10:12 AM, Colin Walters wrote: On Fri, 2013-04-26 at 08:46 -0400, Matthias Clasen wrote: You are not going to get me to buy eagerly into a new installed tests scheme for glib if it means that I have to give up make check. Well, would you be OK with: $ jhbuild make $ gnome-desktop-testing-runner glib I want make distcheck to still run all of my tests, to guarantee that everything works correctly when built from a tarball, not just when built from git. That's going to be a high bar to jump; but I suppose it makes sense to have both during the transition and give downstreams time to teach their build systems about revision control. It forces the code into the you don't have futexes fallback mode, so that we can test that that codepath works, even on machines that do have futexes. We absolutely want to include that as part of the installed tests too. Mmm...but in a true black box model, that would require *installing* /usr/share/glib/gbitlock.c. In the way I have gjs/installed-tests set up, a build of the installed tests, even though it uses the same git repository, is not supposed to use the gjs source code. It's just a way to avoid doubling the number of git repositories. I understand the win of installed tests, but that doesn't mean that make check tests don't have their own set of wins. I want both. Ok, I will work on it sometime this cycle. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Fri, 2013-04-26 at 14:44 +0100, Patrick Welche wrote: On Thu, Apr 25, 2013 at 10:21:47AM -0400, Colin Walters wrote: https://live.gnome.org/GnomeGoals/InstalledTests I'm not entirely sure what sort of tests your framework is designed to contain... Most of what currently lives as make check in the git repositories; see my earlier mail for the characteristics. Have you seen Brian Nitz's work on GNOME accessibility tests that can be automatically run from a tinderbox? Just read the PDF; it's interesting and useful. a11y based tests make a lot of sense. I know apps like pitivi already include their own a11y-based automated tests; it would make sense to me for the apps to contain their own tests, since that would allow them to adapt to their own UI changes rather than having to keep two repositories in sync. Applications and components could then reuse the proposed installed tests spec to describe these tests, and they could then be easily automated. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
Hi, On 04/26/2013 05:01 PM, Colin Walters wrote: On Fri, 2013-04-26 at 10:32 -0400, Dan Winship wrote: I want make distcheck to still run all of my tests, to guarantee that everything works correctly when built from a tarball, not just when built from git. That's going to be a high bar to jump; but I suppose it makes sense to have both during the transition and give downstreams time to teach their build systems about revision control. I'm not sure I follow here. Are you implying that you want to stop making tarballs eventually? Cheers, Emilio ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Fri, 2013-04-26 at 18:49 +0200, Emilio Pozuelo Monfort wrote: Hi, On 04/26/2013 05:01 PM, Colin Walters wrote: On Fri, 2013-04-26 at 10:32 -0400, Dan Winship wrote: I want make distcheck to still run all of my tests, to guarantee that everything works correctly when built from a tarball, not just when built from git. That's going to be a high bar to jump; but I suppose it makes sense to have both during the transition and give downstreams time to teach their build systems about revision control. I'm not sure I follow here. Are you implying that you want to stop making tarballs eventually? Yes. https://mail.gnome.org/archives/release-team/2013-April/msg00038.html ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Sat, Apr 27, 2013 at 1:56 AM, Colin Walters walt...@verbum.org wrote: On Fri, 2013-04-26 at 18:49 +0200, Emilio Pozuelo Monfort wrote: Hi, On 04/26/2013 05:01 PM, Colin Walters wrote: On Fri, 2013-04-26 at 10:32 -0400, Dan Winship wrote: I want make distcheck to still run all of my tests, to guarantee that everything works correctly when built from a tarball, not just when built from git. That's going to be a high bar to jump; but I suppose it makes sense to have both during the transition and give downstreams time to teach their build systems about revision control. I'm not sure I follow here. Are you implying that you want to stop making tarballs eventually? Yes. https://mail.gnome.org/archives/release-team/2013-April/msg00038.html We're not all on that mailing list (I would venture to say only a select few people who are part of the release-team are on that list). Frankly I think it's a really bad idea to expect people to build from git, prepackaged tarballs are much easier to build. What is the point of configure.ac without a tarball ? with no tarball you unconditionally require that the m4s for every dependency, even a soft dependency be present (because the m4s for soft dependencies are required to actually build the configure script). Not to mention, build mechanisms generally use tarballs and prefer them (eg. buildroot, sure you can configure packages to download from git, etc, but it's generally an overhead vs. downloading a nice standard tarball which you know will build as you expect it to, every time). If you want to remove our shell accounts, please provide us with an alternative way of publishing our tarballs (and do give us a heads up on d-d-l before hand, so we can collect anything which we might be keeping in our $(HOME) and $(HOME)/public_html). Cheers, -Tristan ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On 26/04/13 18:26, Tristan Van Berkom wrote:
Not to mention, build mechanisms generally use tarballs and prefer
them
If the tarballs produced by cgit become the recommended thing for
tarball-based build systems to consume - perhaps by having a branch off
to one side that does contain the generated Autotools goo - are they
deterministic, even across git/cgit upgrades (or, potentially, switching
from cgit to some other thing)?
Many tarball-based build systems are of the form download
glib-2.36.0.tar.xz from $MIRROR, then check that its SHA256 is
0123...def (and in those that don't, that's probably a security
vulnerability anyway).
In rpm and dpkg, the tarball is copied into the downstream source
package (rather than being downloaded on-demand from off-site as in,
e.g., BSD ports), but there's still value in the tarball used for glib
2.36.0-1 and glib 2.36.0-7 being exactly the same. (If nothing else,
Debian only keeps one copy per upstream version, to reduce the size of
its package download mirrors from ridiculous to merely very large.)
make distcheck also has value even if you don't care about the
resulting tarball: it tests various things which are not typically done
by maintainers for normal hacking builds, but still ought to work.
(Out-of-tree builds, installation, uninstallation, DESTDIR, alternative
${prefix} and so on; in Telepathy, we also compile with --disable-debug
during distcheck, precisely because maintainers' normal builds would
basically never test that path otherwise.)
S
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
2013/4/26 Tristan Van Berkom t...@gnome.org On Sat, Apr 27, 2013 at 1:56 AM, Colin Walters walt...@verbum.org wrote: On Fri, 2013-04-26 at 18:49 +0200, Emilio Pozuelo Monfort wrote: Hi, On 04/26/2013 05:01 PM, Colin Walters wrote: On Fri, 2013-04-26 at 10:32 -0400, Dan Winship wrote: I want make distcheck to still run all of my tests, to guarantee that everything works correctly when built from a tarball, not just when built from git. That's going to be a high bar to jump; but I suppose it makes sense to have both during the transition and give downstreams time to teach their build systems about revision control. I'm not sure I follow here. Are you implying that you want to stop making tarballs eventually? Yes. https://mail.gnome.org/archives/release-team/2013-April/msg00038.html If you want to remove our shell accounts, please provide us with an alternative way of publishing our tarballs (and do give us a heads up on d-d-l before hand, so we can collect anything which we might be keeping in our $(HOME) and $(HOME)/public_html). As you can see from the subject of my e-mail at [1], the whole discussion is just a proposal at this moment of time, again just a proposal (it didn't go to d-d-l yet, cause we are still discussing all the possible solutions and the first step is usually hearing what the release-team has to say). I'm currently working with Olav Vitters to build a script that will work exactly as ftpadmin does now with a little but big difference: you'll be able to run it locally on your machine. (ideally you will have to rsync the tarball to the server and then install it as you currently do) This way we'll be able to disable logins to master.gnome.org but people will still be able to provide tarballs for their modules. Tools are evolving really fast and honestly I feel that the future will definitely see tarballs being dropped at some point. And that point is not close, I believe. Also as a side note, the logins to people.gnome.org *won't* be disabled, we are talking about logins to master.gnome.org now which is the only machine that has so many logins and ssh keys stored on the whole subsets of systems we host on the GNOME Infrastructure. -- Cheers, Andrea Debian Developer, Fedora / EPEL packager, GNOME Sysadmin, GNOME Foundation Membership Elections Committee Chairman Homepage: http://www.gnome.org/~av ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Fri, 2013-04-26 at 12:56 -0400, Colin Walters wrote: On Fri, 2013-04-26 at 18:49 +0200, Emilio Pozuelo Monfort wrote: Hi, On 04/26/2013 05:01 PM, Colin Walters wrote: On Fri, 2013-04-26 at 10:32 -0400, Dan Winship wrote: I want make distcheck to still run all of my tests, to guarantee that everything works correctly when built from a tarball, not just when built from git. That's going to be a high bar to jump; but I suppose it makes sense to have both during the transition and give downstreams time to teach their build systems about revision control. I'm not sure I follow here. Are you implying that you want to stop making tarballs eventually? Yes. https://mail.gnome.org/archives/release-team/2013-April/msg00038.html At least on Gentoo there is partially existing infrastructure but it is not considered superior to tarballs. The collision attack on git is possible, especially when the build is automated[1] and presumably by not closely watching user, while tarballs have their hash distributed by another channel on Gentoo. While for main releases in portage it is not a problem as probably the tarballs would be made by gnome team and uploaded they pose more of a problem for the overlays (sort of additional repos - new packages/unstable versions are usually found there) or bugzilla (another place where people are posting ebuilds). I would presume other source distribution (Nix, AUR in Arch) have similar problems. Additionally currently the Vala programs can be compiled with VALAC=/bin/true and depend entirely on generated sources which are not kept in git (as they are autogenerated during make distcheck). That came handy during the Gnome 3.8 transition as there are several packages that requires older version of Vala, which vapigen is not compatible with gobject-introspection 3.8. Such scheme would not be possible with git tags as the sources wouldn't exist. This would be problem for Vala among others as there would be no way to bootstrap compiler except by another Vala compiler. The same problem is for any files which are meant to not be built (always) by user but are not kept in git - including say autotools files. Requiring each user to run autogen would make the compilation longer (in addition to possible errors due to changes in autoconf/automake). I understand why access people what to restrict access to master.gnome.org but wouldn't replacing shell access with something else be sufficient (or possibly restricting it to scp possibly with special ui for verification without general purpose shell or tarball autogeneration)? Best regards [1] http://joeyh.name/blog/entry/sha-1/ ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Fri, Apr 26, 2013 at 9:44 PM, Colin Walters walt...@verbum.org wrote: Anyways...dropping tarballs is not going to happen soon, but I'm happy to consider what we need to do to lay the foundations now. More short-term, would it make sense to leverage the build servers to run make distcheck when a new tag is pushed and take care of uploading the generated tarballs? (Though there's some temptation for maintainers to be lazy and create a tag without making sure that distcheck actually passes ...) ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Fri, 2013-04-26 at 22:56 +0200, Florian Müllner wrote: On Fri, Apr 26, 2013 at 9:44 PM, Colin Walters walt...@verbum.org wrote: Anyways...dropping tarballs is not going to happen soon, but I'm happy to consider what we need to do to lay the foundations now. More short-term, would it make sense to leverage the build servers to run make distcheck when a new tag is pushed and take care of uploading the generated tarballs? Maybe. The make dist part would probably be relatively doable to automate for most modules that don't have many sysdep type dependencies. But the check part is harder because we'd need a well defined, reproducible environment for tests which goes back to the installed tests design. Currently it's completely up to the individual module maintainers what's running on *their* laptop when they type make distcheck. There's some modules out there that I'm sure have only ever been distchecked on Ubuntu or OpenSUSE for example. (Though there's some temptation for maintainers to be lazy and create a tag without making sure that distcheck actually passes ...) Yeah, that's an issue. But there's another related and also important one: Remember, tests don't pass or not in an abstract sense; the only pass or not *in a specific environment*. It's entirely possible that distcheck passes on *your* laptop but not on the build servers. For example, you have a 32 bit installation, the build server is 64 bit. This is the kind of issue that can be a tremendous pain to debug because then you want access to the environment so you can run gdb, etc. Years of thought on this made me realize that it'd actually be far better if the build server just spit out binaries as fast as it could, and the tests ran automatically, continuously after that, tagging builds which passed tests. If you need to debug something, you *can* download the binaries straight from the build server (buildmaster), and debug those locally. Which is how the gnome-ostree build system works. But that only came after several failed build systems attempting to do (part of) something like what you're describing. Going back to the top though, make dist seems somewhat plausible for automation, the biggest question is what the environment is (jhbuild on top of Ubuntu/Fedora?). ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
Hey Colin, On Thu, 2013-04-25 at 10:21 -0400, Colin Walters wrote: See: https://live.gnome.org/GnomeGoals/InstalledTests I'm in the process of transitioning to that page from https://docs.google.com/document/d/1DXgGEKTbC4ed1DFW3Mu-48TpHtq-xdfhDtwoUOWWcHg/edit Comments appreciated; I have prototype branches of gnome-ostree, gnome-desktop-testing, and gjs that implement this, but it's all subject to change. What are the requirements in terms of what tests are allowed to do? I'm guessing they shouldn't touch the running environment, create files in the user's home directory, etc. Are we allowed to access the network for example? Do we have (makefile) infrastructure to allow GTests to run both uninstalled (during make *check) and installed? Can we access the network? Cheers ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Thu, 2013-04-25 at 16:30 +0200, Bastien Nocera wrote: What are the requirements in terms of what tests are allowed to do? I'm guessing they shouldn't touch the running environment, That's a good question. The gnome-ostree testing infrastructure uses qemu with a copy-on-write qcow2 disk snapshot for all tests; they're run as an autologged in user with an active desktop session. It'd absolutely be possible to have tests which mutate the environment, or even run as root and possibly trash the system. But the primary thing I'm focusing on now is transitioning most of GNOME's set of tests which tend to have these characteristics: * Run as non root * Assume the presence of a logged in session (or sometimes autocreate one with Xvfb/dbus-launch) * Are nondestructive Almost all of the GLib, gjs, and pygobject test cases are of that form; by having them be installed, builders and integrators can fully automate running them however often they choose. Though you've got me thinking: maybe instead of the /usr/bin/foo-installed-test naming scheme, we should have /usr/share/installed-tests/foo.desktop. Desktop files by their nature already: 1) Are what we use to launch binaries 2) Have metadata If other people like that idea, I'll look at reworking the testing code to use .desktop files. create files in the user's home directory, etc. Are we allowed to access the network for example? I don't think we should write network-dependent tests at this time; they'd be a huge source of unreliability. Now it'd obviously be very cool to have e.g. a static copy of a wikipedia page hosted in GNOME infrastructure, and have a test which automates running Epiphany against that page. But let's learn to walk before we run =) Do we have (makefile) infrastructure to allow GTests to run both uninstalled (during make *check) and installed? Not at this time; that'd be nice, but I think the jhbuild model mostly obviates the need for uninstalled tests, because jhbuild by nature is a sandboxed environment independent from the underlying system. cd ~/src/glib jhbuild make # note: runs make install ~/build/bin/glib-proxy-installed-test So if you broke glib, you still have the underlying system to fall back on. Now for things like NetworkManager that jhbuild is incapable of handling, I can certainly imagine how uninstalled tests would be very convenient for developers. And it would likely make sense for some of those tests to support running them both installed and uninstalled. But the point of the gnome-ostree model is precisely to test those types of things that have historically had very bad testing coverage because jhbuild is incapable of handling them, and sudo make install and hacking up distribution packages also sucks. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On 04/25/2013 10:21 AM, Colin Walters wrote: See: https://live.gnome.org/GnomeGoals/InstalledTests Installed tests go in /usr/bin/testname-installed-test. That seems unnecessary, since they're not run as part of ordinary system operation, and it will pollute autocompletion. I think it would be better to have some other standard location for them. Also, they need to include the package name (either as a path component, or as part of the test name), to avoid conflicts. (eg, glib and libsoup both have a proxy-test.) We could install both the test binaries and a Makefile into, eg, /usr/libexec/$pkgname/tests/, and then you can just run make -C /usr/libexec/$pkgname/tests check to run all of them. (And maybe have a wrapper for that so you just type run-installed-tests glib.) They should exit with code 0 on success, and any other code on failure. I recently learned that automake's test harness supports two special exit statuses: 77 for skipped (it's not currently possible/meaningful to run the test, eg, because of a missing run-time dependency, or because this system doesn't support the feature being tested), and 99 for hard failure (something went wrong and the test can't continue, but the failure is not related the thing that's being tested). Skipped is definitely useful. Hard failure is useful in theory (eg, disk full), but less useful in practice since it would be a pain to distinguish those sorts of failures in most places. -- Dan ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On 25/04/13 15:45, Colin Walters wrote:
Though you've got me thinking: maybe instead of
the /usr/bin/foo-installed-test naming scheme, we should
have /usr/share/installed-tests/foo.desktop. Desktop files by their
nature already:
1) Are what we use to launch binaries
2) Have metadata
I like this idea, and I imagine it wouldn't be hard to hook this into
other test harnesses (e.g. describe each test in that directory as a
Debian autopkgtest, or get Jenkins CI to run them, or whatever).
Having these test binaries on $PATH seems bad, particularly if they're
allowed to damage $HOME when run without precautions, so having them
canonically go in ${pkglibexecdir} (with the exact location being an
implementation detail because of the use of .desktop files) would
probably be better.
* Run as non root
* Assume the presence of a logged in session (or sometimes autocreate
one with Xvfb/dbus-launch)
* Are nondestructive
I think the default should be that tests are non-destructive, run as
non-root, do not require a session, do not require networking and so on;
if any of these are not true, they should be declared via something
analogous to autopkgtest's Restrictions.
The currently-defined Restrictions in autopkgtest are rw-build-tree,
breaks-testbed, needs-root and build-needed. The two build ones are not
relevant for installed tests. GNOME could use needs-root, breaks-testbed
(if it breaks the system) and breaks-home (if it trashes $HOME), perhaps?
S
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Thu, 2013-04-25 at 16:41 +0100, Simon McVittie wrote:
This general plan looks really good to me. I think it'd help us really
tighten up quality at the boundaries of our projects.
On 25/04/13 15:45, Colin Walters wrote:
Though you've got me thinking: maybe instead of
the /usr/bin/foo-installed-test naming scheme, we should
have /usr/share/installed-tests/foo.desktop. Desktop files by their
nature already:
1) Are what we use to launch binaries
2) Have metadata
I like this idea, and I imagine it wouldn't be hard to hook this into
other test harnesses (e.g. describe each test in that directory as a
Debian autopkgtest, or get Jenkins CI to run them, or whatever).
Having these test binaries on $PATH seems bad, particularly if they're
allowed to damage $HOME when run without precautions, so having them
canonically go in ${pkglibexecdir} (with the exact location being an
implementation detail because of the use of .desktop files) would
probably be better.
Additionally, I wouldn't want hundreds of installed tests block shell
tab-completion from being useful.
* Run as non root
* Assume the presence of a logged in session (or sometimes autocreate
one with Xvfb/dbus-launch)
* Are nondestructive
I think the default should be that tests are non-destructive, run as
non-root, do not require a session, do not require networking and so on;
if any of these are not true, they should be declared via something
analogous to autopkgtest's Restrictions.
The currently-defined Restrictions in autopkgtest are rw-build-tree,
breaks-testbed, needs-root and build-needed. The two build ones are not
relevant for installed tests. GNOME could use needs-root, breaks-testbed
(if it breaks the system) and breaks-home (if it trashes $HOME), perhaps?
We wouldn't need to support these types of tests immediately, but I do
think it'd be useful to plan for them. There are classes of tests which
I wouldn't want to run on my live system but which I would want run
automatically on a test system. Eg, a test that includes switching to
airplane mode and back to check that the system dims out network-related
functionality and re-enables it once the network is back up.
-Travis
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: New GnomeGoal proposal: InstalledTests
On Thu, 2013-04-25 at 16:41 +0100, Simon McVittie wrote: The currently-defined Restrictions in autopkgtest are rw-build-tree, breaks-testbed, needs-root and build-needed. The two build ones are not relevant for installed tests. GNOME could use needs-root, breaks-testbed (if it breaks the system) and breaks-home (if it trashes $HOME), perhaps? For needs-root I am thinking now that the best way for these to work is to quite simply install systemd units, sysv init scripts, upstart jobs or whatever. Disabled by default, obviously. This model would be particularly good for systemd units and upstart jobs because they already have expressive ways of hooking into the OS boot order, which a lot of these needs-root tests are going to want. We'd just need a naming scheme or something so that a tool could turn them on programatically. How about installed-test-$PKGNAME-$TESTNAME ? As for breaks-home, let's add that on later; it's easily supported in the gnome-ostree test framework, but obviously it's not something you'd want in jhbuild =)Although - with fast user switching, this could actually work; jhbuild should probably support that. Otherwise these are great, constructive comments, keep them coming! Thanks all who replied so far. I've adapted the code to use .desktop file like files (currently called .testmeta), and updated both the wiki page: https://live.gnome.org/GnomeGoals/InstalledTests#preview and the code: https://git.gnome.org/browse/gnome-desktop-testing/commit/?id=ed967e21209930356814af187b819a95258478d0 https://git.gnome.org/browse/gjs/commit/?h=wip/installed-testsid=f7e4b445f085be31f9f409d0a968398b04f27859 Planning to push this live in the next day or so and we'll see how it works out for gjs. If any project for whom gnome-ostree is currently doing continuous integration wants to add installed tests, please ping me. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
