On 29/12/2010, at 2:37 AM, Dierk König wrote: > we shouldn't call this 'testing'. > > If anything, it is 'excercising' the application. > > As far as gradle is concerned, I don't understand this thread.
Testing, or exercising, the application is such a common activity at each stage of the application lifecycle. As this thread makes clear, it's about much more than just running some unit tests when you build your jar. To me, the relevance to Gradle is that we want to capture the various testing patterns, so that people can reuse them easily in their builds. At its most basic, Gradle is just a bunch of build patterns implemented as plugins. And there's no reason why we should stop with unit tests when other types of testing also need to be automated. > If there is any code that should be called without dependency, > one can simply wrap it in a task that has no dependency and call that task. > > Dierk > > > Am 28.12.2010 um 09:29 schrieb [email protected]: > >> Just another case... >> >> Functionally testing ajax heavy sites (with Geb of course), or any >> concurrent type code. With the ajax case, I find that I don't have much >> faith the *tests* are good until I have run them a few times to get the >> timeouts right. >> >> On 27/12/2010, at 6:47 AM, Adam Murdoch <[email protected]> wrote: >> >>> >>> On 24/12/2010, at 6:56 AM, Peter Niederwieser wrote: >>> >>>> >>>> >>>> Adam Murdoch-3 wrote: >>>>> >>>>>> Sometimes there are external factors. >>>>> >>>>> I guess I was after something a bit more concrete. >>>>> >>>> >>>> A few examples that come to my mind: >>>> >>>> - A test that generates random inputs (property-based testing, testing for >>>> deadlocks, etc.) >>>> - A test that reads inputs/outputs from an external Excel sheet >>>> - An integration test hitting a remote (test) database >>>> - An acceptance test that runs against a live website >>>> >>> >>> I like these examples. It feels like there are a few aspects here: >>> >>> * Tests which use resources other than those on the classpath, such as the >>> excel sheet, database, or web site. >>> >>> At the moment, we model these as inputs and outputs of the test task. And >>> only for local files. But it might be interesting to model other sorts of >>> resources - such as remote files, or database instances or web >>> applications. Then, we can skip a given test if the resources it uses have >>> not changed since last time the test was executed. >>> >>> Not sure how useful this is for incremental testing. But it is useful for >>> things such as test setup and tear down, where Gradle can make sure the >>> resources are deployed before the test and cleaned up after the test. And >>> for reusing the tests in different contexts. A test can declare a >>> dependency on a particular web app, and Gradle can inject the right config >>> into the test: an embedded deployment for a dev build, a test deployment in >>> staging for the ci build, and the production web app as a smoke test for >>> the deployment build. >>> >>> >>> * Tests which have some non-deterministic element >>> >>> Each test execution is a data point that increases confidence in the system >>> under test. In a sense, all tests are like this to some degree. >>> >>> To me, running the test task multiple times from the command-line to >>> increase confidence is in itself a test case, and is probably worth >>> capturing in the automated test suite somehow. >>> >>> One option is to define this in the build: we might introduce the concept >>> of a test suite, and allow you to specify things such as how many times a >>> given test or suite should be executed for us to have confidence that the >>> test has 'passed'. We might add other constraints too: this test must run >>> repeatedly for 8 hours, this test must run on a machine with at least 2 >>> physical cores, this test must run repeatedly at the rate of 30/minute, >>> this test must run concurrently on at least 4 difference machines, etc. >>> >>> These constraints would be useful for other use cases to, such as: this >>> test must run under java 5, this test must run on a linux machine, this >>> test must run on each of a windows, linux and mac machine, this test must >>> run on a machine with oracle installed, this test must run against each of >>> oracle, postgres, mysql, etc. >>> >>> Regardless, I think it is an important point you make that tests are not >>> always deterministic and may need to execute multiple times. We should >>> capture this some how. >>> >>> >>> * Tests which run at multiple points in the application lifecycle. >>> >>> For example, some acceptance tests which run in the developer pre-commit >>> build, the ci build, and the deployment build. You might inject a different >>> web app at each stage, and you might add difference constraints at each >>> stage. >>> >>> I wonder if you might also want different up-to-date strategies at each >>> stage. For a dev build, I don't want to run a set of acceptance tests if I >>> haven't changed the system under test since I last ran them (and they've >>> passed according to whatever constraints have been specified). But, for a >>> deployment build, I might want to specify that the acceptance tests must >>> always be run. >>> >>> >>> -- >>> Adam Murdoch >>> Gradle Developer >>> http://www.gradle.org >>> CTO, Gradle Inc. - Gradle Training, Support, Consulting >>> http://www.gradle.biz >>> > -- Adam Murdoch Gradle Developer http://www.gradle.org CTO, Gradle Inc. - Gradle Training, Support, Consulting http://www.gradle.biz
