Hey,
I'll come up with an initial spec for this, please give feedback. The spec
lives here:
https://github.com/gradle/gradle/blob/master/design-docs/test-selection.mdI'm
including the content here because some formatting is not good.
Use case: legacy integration tests are slow, need a way to execute a single
test method, from command line.
Options:
1. Change the semantics of test.include so that it supports syntax like
{ant-file-pattern}#{testMethodName}, e.g. **/FooTest#someMethod
Test implements PatternFilterable, which declares the exact behavior of
include / exclude. In order to implement this option, we would need to
introduce a mildly breaking change and make Test no longer extend
PatternFilterable. It is kind of a mildly breaking because we would keep
all the existing public methods in the Test type. If users extends Test
task, they would probably need to recompile. For command line, we introduce
--include option for the Test task. Pros: api remains small, Cons: breaking
change.
2. Introduce new dsl (various ideas lumped together):
test {
selection {
//using 'include' wording here could make it confusing with test.include
select "**/Foo*#someOtherMethod", "**/Foo*#someMethod"
selections = []
//in/out
in "**/Foo*#someOtherMethod", "**/Foo*#someMethod"
out "**/Foo*#someOtherMethod", "**/Foo*#someMethod"
//if we deprecate test.include, we could do
include "**/Foo*#someOtherMethod", "**/Foo*#someMethod"
includes = []
//keep method selection separate
includeMethod
includeMethods = []
include {
method
methods = []
}
//some other elements we could add in future
exclude
excludes = []
unselect
unselections = []
include {
descendantsOf 'com.foo.SomeBaseClass'
annotatedWith 'com.foo.Slow'
matching { descriptor, testClass ->
//...
}
}
}
}
Then add consistent commandline support, e.g.
gradle test --select **/Foo.java#someTest
3. It would be good to consider the command line interface when designing
the dsl because they need to be consistent. The convenient command line
should support:
- selecting class + method in one option
- allow wildcards for classes (wildcards for methods are not practical to
implement)
- support both, file separators '/' or fqn separators '.'
--select com/bar/**/Foo.java#someTest
--select Foo#someTest
--select com.foo.bar.Foo#otherTest
4. Here's api I was thinking about as a first story, please give feedback:
test {
//good old include
include '**/*SomeIntegrationTest'
//new stuff
selection {
include {
method 'method1', 'method2'
methods = []
}
}
}
Plus, a convenience method directly in the test task, so that all above
could be inlined into:
test {
select '**/*SomeIntegrationTest#method1,method2'
//or:
select '**/*SomeIntegrationTest#method1',
'**/*SomeIntegrationTest#method2'
}
Then, command line support could be:
gradle test --select **/*SomeIntegrationTest#method1,method2
I would also deprecate test.single property
Cheers!
On Fri, Nov 15, 2013 at 4:23 AM, Adam Murdoch
<adam.murd...@gradleware.com>wrote:
>
> On 13 Nov 2013, at 7:36 am, Szczepan Faber <szczepan.fa...@gradleware.com>
> wrote:
>
> Heya,
>
> I'm keen on solving this use case: developer in a large app has tons of
> long running tests. He wants to run only a *single* test from the command
> line (e.g. we currently support *all* tests from a *single* test class via
> test.single).
>
> There's also a pull request: https://github.com/gradle/gradle/pull/193that
> shows that the community is also keen on getting it sorted out.
>
> How do we want approach this?
>
> 1. Make the test.includes / test.excludes more robust and support some
> kind of notation like "SomeTest#someMethod". This is kind of awkward
> because Test extends PatternFilterable that clearly defines what are
> includes and excludes (they are not test methods, they are ant file
> patterns). On the plus side, this would work with our existing means of
> specifying includes/excludes (existing DSL, existing test.single property)
>
> 2. Add some new API for this, for example:
>
> test.selection.include ...
>
> It might be good to put the test selection api behind some new DSL
> layer/object (e.g. 'selection') because it can potentially grow:
> include/exclude tests by some custom criteria, e.g. test class hierarchy,
> etc. BTW. for the latter we have use cases in our own codebase (cross
> version tests now are picked up by naming convention but it would be nicer
> if they were picked up by parent class, etc.).
>
> If we go down this path, we need to add command line support for it and
> perhaps consider deprecation of the existing include / exclude.
>
> Do we have some other options? Thoughts?
>
>
> I’d go with option #2. The current includes and excludes are really
> intended to specify where to look for tests. Option 2 adds another filter
> over this to specify some criteria to use for selecting tests (beyond where
> they physically live).
>
> I’d also add command-line support and deprecate the system property.
>
>
> --
> Adam Murdoch
> Gradle Co-founder
> http://www.gradle.org
> VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
> http://www.gradleware.com
>
>
>
>
--
Szczepan Faber
Principal engineer@gradle; Founder@mockito
