On 13/09/2011, at 9:18 AM, Szczepan Faber wrote: > Cool feedback - good ideas. No doubt we will iterate and improve the way we > test the daemon. > > So far I've been leaning towards below strategy for daemon testing (it's > personal and very biased - you've been warned :). > > 1. I'd like to have functional tests that specifically target the daemon > infrastructure and are decoupled from gradle builds (for better > maintainability).
What I'm suggesting wouldn't be coupled to the build. Very few of the integration tests require you to run anything in the build before you can run them (there are a handful of exceptions, that do things like test the documentation). It would be the same with this integration test. You'd simply make changes in the code and then run the test. > This probably means we need to aim to decouple daemon infrastructure (as much > as possible) from 'executing gradle builds'. Example of such test: > DaemonFunctionalTest (gives some headache but we will iterate until it's > perfected). I'm not very interested in running those tests in 'embedded > daemon' mode because they are focused on the daemon infrastructure and the > essence of this infrastructure is cross-process synchronization and > communication. > > 2. Integration tests that actually run gradle builds and use our precious > GradleExecuter. Example of such test: EnvironmentVariablesIntegrationTest > (very interesting especially when executed with the Daemon!). Other examples: > all other integ tests. When you have such an integration test, which you can run and debug from the IDE without running the build, then DaemonFunctionalTest doesn't add any value. If you look at the test cases in DaemonFunctionalTest, they are exactly the same cases we need in the integration test. So, rather than duplicate all the test cases in a separate integration test and keep them in sync for no benefit, I'm suggesting we simply repurpose DaemonFunctionalTest into a real integration test, but keep the ability to run it from the IDE. > > 3. unit tests > > The combination of those 3 testing strategies gives excellent coverage and > good maintainability. The idea is something that clarified for me after some > time of working with the daemon code. Initially I started differently: using > daemon distribution executor to run mini-builds, synchronizing builds via > files, etc. However, that didn't work very well (high maintenance, slow > cycles, difficult debugging, etc.) I think a much better strategy is to make the integration test cases reusable in different contexts. This is the approach we've taken with all the other integration test. It works very well, I believe. For example, we got a daemon test suite for free by using this approach. I don't see how the daemon is different enough to warrant a different approach. It simply sounds like we're just missing a fixture or two to make it easy to write reusable tests. > > Let's talk about daemon testing sometime today or tomorrow. I have couple of > questions. > > Cheers! > > On Mon, Sep 12, 2011 at 12:18 AM, Adam Murdoch <[email protected]> > wrote: > > On 11/09/2011, at 6:09 PM, [email protected] wrote: > >> Branch: refs/heads/master >> Home: https://github.com/gradle/gradle >> >> Commit: 05308efa20db9712f2def074e9e2cb3577dbb86b >> >> https://github.com/gradle/gradle/commit/05308efa20db9712f2def074e9e2cb3577dbb86b >> Author: Szczepan Faber <[email protected]> >> Date: 2011-09-11 (Sun, 11 Sep 2011) >> >> Changed paths: >> M >> subprojects/integ-test/src/integTest/groovy/org/gradle/integtests/daemon/DaemonFunctionalTest.groovy > > I think this should become a real integration test. If you look at the method > names, they reflect features we want Gradle to have regardless of which > interface you use: daemons expire, stops all daemon, starts a new daemon if > daemon is busy, etc. This is stuff we want coverage for, so it would be nice > if we can reuse this test in different contexts. > We should introduce a daemon integ test fixture, with 2 implementations: an > 'embedded' mode, which bolts directly onto DaemonClient and DaemonRegistry, > and a 'forking' mode, which runs real builds to start the daemon, and uses > 'gradle --stop' to stop the daemons. > It would need to use the DaemonRegistry for now, to probe the status of the > daemons. But this is something we want to add to the command-line, at some > point, eg http://issues.gradle.org/browse/GRADLE-1462. > > We should also get rid of the Sleep command, and just run a build that sleeps > for the appropriate amount of time. Plus find a way of synchronising between > the test and the test build that doesn't involve hard-coded sleeps, perhaps > using a marker file. > > > -- > Adam Murdoch > Gradle Co-founder > http://www.gradle.org > VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting > http://www.gradleware.com > > > > > -- > Szczepan Faber > Principal engineer@gradleware > Lead@mockito -- Adam Murdoch Gradle Co-founder http://www.gradle.org VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com
