On 28/12/2010, at 7:29 PM, [email protected] wrote: > Just another case... > > Functionally testing ajax heavy sites (with Geb of course), or any concurrent > type code. With the ajax case, I find that I don't have much faith the > *tests* are good until I have run them a few times to get the timeouts right.
This is an interesting one. Tuning timeouts in tests is a real pain. In my experience, they're a common cause of false build breakages. And they're of questionable value because, in practise, people just keep making them larger and larger until the build stops breaking. It would be nice if there were some way for the tools to discover what the timeouts should be. For example, have some way to run a test a bunch of times on a bunch of machines to figure out what the distribution of execution times for a given operation. After that, the test can use these values, and break if there's a regression in execution time. On a timeout failure, we could also re-run the test a few times to discover if there is a real regression in the execution time. These behaviours might also change depending on where the test is running, and which stage of the application lifecycle it is running. For example, when running on a developer machine as part of the pre-checkin build, the tests might allow some variation in execution time. When running in the CI build against a production-like environment, the test might have a hard limit on execution time, and may also be run a number of times. There'd also need some way to visualise the execution time distributions for a given operation. Of course, most of this is a problem for the testing tool to solve. But, Gradle could do a better job of interacting with the testing tool to dynamically schedule tests based on their execution results, and to provide better reporting and ways to inject configuration into tests and the testing tool. > > On 27/12/2010, at 6:47 AM, Adam Murdoch <[email protected]> wrote: > >> >> On 24/12/2010, at 6:56 AM, Peter Niederwieser wrote: >> >>> >>> >>> Adam Murdoch-3 wrote: >>>> >>>>> Sometimes there are external factors. >>>> >>>> I guess I was after something a bit more concrete. >>>> >>> >>> A few examples that come to my mind: >>> >>> - A test that generates random inputs (property-based testing, testing for >>> deadlocks, etc.) >>> - A test that reads inputs/outputs from an external Excel sheet >>> - An integration test hitting a remote (test) database >>> - An acceptance test that runs against a live website >>> >> >> I like these examples. It feels like there are a few aspects here: >> >> * Tests which use resources other than those on the classpath, such as the >> excel sheet, database, or web site. >> >> At the moment, we model these as inputs and outputs of the test task. And >> only for local files. But it might be interesting to model other sorts of >> resources - such as remote files, or database instances or web applications. >> Then, we can skip a given test if the resources it uses have not changed >> since last time the test was executed. >> >> Not sure how useful this is for incremental testing. But it is useful for >> things such as test setup and tear down, where Gradle can make sure the >> resources are deployed before the test and cleaned up after the test. And >> for reusing the tests in different contexts. A test can declare a dependency >> on a particular web app, and Gradle can inject the right config into the >> test: an embedded deployment for a dev build, a test deployment in staging >> for the ci build, and the production web app as a smoke test for the >> deployment build. >> >> >> * Tests which have some non-deterministic element >> >> Each test execution is a data point that increases confidence in the system >> under test. In a sense, all tests are like this to some degree. >> >> To me, running the test task multiple times from the command-line to >> increase confidence is in itself a test case, and is probably worth >> capturing in the automated test suite somehow. >> >> One option is to define this in the build: we might introduce the concept of >> a test suite, and allow you to specify things such as how many times a given >> test or suite should be executed for us to have confidence that the test has >> 'passed'. We might add other constraints too: this test must run repeatedly >> for 8 hours, this test must run on a machine with at least 2 physical cores, >> this test must run repeatedly at the rate of 30/minute, this test must run >> concurrently on at least 4 difference machines, etc. >> >> These constraints would be useful for other use cases to, such as: this test >> must run under java 5, this test must run on a linux machine, this test must >> run on each of a windows, linux and mac machine, this test must run on a >> machine with oracle installed, this test must run against each of oracle, >> postgres, mysql, etc. >> >> Regardless, I think it is an important point you make that tests are not >> always deterministic and may need to execute multiple times. We should >> capture this some how. >> >> >> * Tests which run at multiple points in the application lifecycle. >> >> For example, some acceptance tests which run in the developer pre-commit >> build, the ci build, and the deployment build. You might inject a different >> web app at each stage, and you might add difference constraints at each >> stage. >> >> I wonder if you might also want different up-to-date strategies at each >> stage. For a dev build, I don't want to run a set of acceptance tests if I >> haven't changed the system under test since I last ran them (and they've >> passed according to whatever constraints have been specified). But, for a >> deployment build, I might want to specify that the acceptance tests must >> always be run. >> >> >> -- >> Adam Murdoch >> Gradle Developer >> http://www.gradle.org >> CTO, Gradle Inc. - Gradle Training, Support, Consulting >> http://www.gradle.biz >> -- Adam Murdoch Gradle Developer http://www.gradle.org CTO, Gradle Inc. - Gradle Training, Support, Consulting http://www.gradle.biz
