On 26/08/2011, at 6:07 PM, Luke Daley wrote:
>
> On 26/08/2011, at 6:45 AM, Adam Murdoch wrote:
>
>> Hi,
>>
>> I'm not sure I like moving (some of) core/src/test out into a separate
>> project. As a general pattern, I think I would much rather that the test
>> fixtures exported by a project live in that project, rather than in a
>> separate project, and are modelled as a separate publication.
>>
>> That is, right now, we have the following dependencies:
>>
>> core test -> internalTesting main -> core main
>>
>> I'd much rather:
>>
>> core test -> core testFixtures -> core main.
>
> core-test is going to have a compile dependency on core-main as well though,
> so it's not quite that linear (unsure if that's relevant here).
>
>> where 'core testFixtures' are the core-specific test fixtures, such as
>> classes like HelperUtil, AbstractTaskTest, AbstractTaskSpec. This doesn't
>> include int testing fixtures such as GradleExecutor, and probably doesn't
>> include general things such as TemporaryFolder, TestFile, and so on.
>
> Makes sense. If you are compiling against core, then you'll likely want to
> compile against coreTestFixtures.
>
> However, at some point we'll probably want to publish a core-fixtures jar of
> public utilities for this and we may want to actually have tests for these
> test fixtures. This could be the same for any plugin, e.g. we might want to
> publish a maven-fixtures jar for unit test fixtures for developers who
> integrate with/use types from the maven plugin(s).
>
> The same could be said for integration tests. There may be a
> maven-integ-fixtures that provides an in memory http server for setting up
> repos for integration tests.
>
> I think this _potentially_ leaves us with the following groups of source for
> a project
>
> main
> test
> testFixtures
> integTest
> integTestFixtures
>
> If we do want to have public test fixtures that might mean another four
> (names aside):
>
> publicTestFixtures
> publicTestFixturesTest
> publicIntegTestFixtures
> publicIntegTestFixturesTest
I think there's no question all these categories of code will exist. However,
I'm not sure we necessarily want to split each category out into its own source
set. We do have some other dimensions we can use to categorise code, and we
could use a mix of these:
* Split into projects
* Split into source sets
* Split into package hierarchies
* probably some others too
I'm not sure what a good scheme would be, yet. I think we could simplify this a
bit for now, let it soak for a while and see what we come up with.
I'd suggest something like this, for now:
* main source set -> any public code, regardless of type.
* testFixtures source set -> internal test fixtures, regardless of type.
* test source set -> unit tests for any code used outside the project (ie main
and test fixtures)
* integTest source set -> int tests for any code used outside the project
One benefit of splitting out the tests and test fixtures is that a change to,
say, a unit test in core won't go invalidating the unit and integration tests
in all the other projects.
Another benefit of splitting out test fixtures is that we can reuse these
internal test fixtures in our other Gradleware work. So, 'internal' above means
the same thing as 'internal' for APIs: it is primarily intended to be used
inside Gradle, but you can use it if you like. Just keep in mind it's unstable
and may change at any time.
We'd keep the internalTesting project, it would contain only testFixtures.
Not sure what to do with the integ test fixtures. The question is whether
integration testing is conceptually part of core or not. It feels to me like it
is a separate enough concern to warrant its own project. Given this, we
wouldn't necessarily need an integTestFixtures source set, as the integTest
project would have:
* main -> public integ test fixtures
* testFixtures -> internal integ test fixtures
* test -> unit tests for the fixtures
* integTest -> integ tests for the fixtures
>
>
> This is starting to look like too much, but all of these (apart from main)
> would be optional. I wouldn't expect many plugins to expose public utilities
> for integration testing but it could definitely happen (and should in some
> cases).
>
> Also, I'd rather have this source set explosion and very carefully control
> what's public than try and jam them together (but I guess having so many
> projects may be burdensome in the IDE).
>
>> One problem - apart from the conceptual awkwardness - with having a
>> project's test fixtures in a separate project is that it blows our 'must be
>> able to import into eclipse' story right out of the water, as the eclipse
>> plugin will map these to:
>>
>> core -> internalTesting -> core
>>
>> which, of course, eclipse can't handle.
>
> If we move the integration testing harness out into its own project (as I
> think you said we should), won't we have the same problem?
We will. This is why my three options below all had a project's integTest
source set mapped to a separate eclipse project.
I think, if we tried, we could end up with something like:
compile:
integTest main/fixtures
-> toolingApi main
-> wrapper main
(currently toolingAPI drags in core, but shouldn't)
int test compile (using scala as an example):
scala int test ->
-> scala main
-> plugins main
-> core main
-> integTest main/fixtures
int test runtime (forking mode):
scala int test
-> integTest main/fixtures
-> toolingApi main
-> wrapper main
-> gradle dist (which happens to contain scala main)
int test runtime (embedded mode):
scala int test
-> scala main
-> plugins main
-> core main
-> integTest main/fixtures
-> toolingAPI main
-> wrapper main
-> launcher main
-> core main
-> coreImpl main
-> ui main
The basic idea is that integTest fixtures drag in only the tooling API and
launcher at runtime. These would then make use of whatever plugins they can
find on the classpath at runtime. Currently, the integ test fixtures drag in
all projects at runtime, which means a huge cycle.
A project would automatically pick up any plugins which it has a
compile/runtime dependency on, and the project can add in more by adding an
integTest compile/runtime dependency.
A benefit of this approach is that a change to, say, the announce plugin, does
not invalidate the integration tests for, say, c++. And that to run the int
tests for the ide plugin, I don't need to build the docs.
I think this works, except if any of integTest's runtime dependencies need to
have integration tests (eg core, tooling API, or ui).
The (perhaps simpler) alternative, is that the integTest source set of every
project is mapped to a separate ide project. Then we can have whatever
dependencies we like (including the above, of course).
>
>> Another issue (for me) is that idea treats the test fixtures as production
>> code, not test code. This breaks things such as search for usages.
>
> So IDEA knows that the test source sets etc. are not “production”?
>
> Does this also mean that for the “public” test fixtures we'd need to have
> them as separate projects so IDEA knows they are production code?
They'd just need to be in a source directory that we can mark as a production
source dir. This might mean that they live in the main source set of the same
project, or in a separate source set, or in a separate project.
>
>> I think there are some better options:
>>
>> 1. We push the separate projects option even further, and bust up every
>> project into 4 separate projects: main, testFixtures, tests, and integTests.
>>
>> 2. Test fixtures live in the owning project, and we map each source set to a
>> separate eclipse project, so we end up with -main, -test, and -integTests
>> eclipse projects. Possibly combine this with adding a testFixtures source
>> set as well.
>
> I don't understand the last sentence. Does that just mean a -testFixtures
> project (just like -integTests) or something else?
>
> This seems like the winner to me.
>
>> 3. Test fixtures live in the owning project, and we map the integTest source
>> set of each project to a separate eclipse project. Possibly combine this
>> with adding a testFixtures source set as well.
>>
>> What is currently 'internalTesting' would split into 3 pieces: the general
>> purpose test fixtures, the core project's test fixtures, and the integration
>> testing fixtures. These would be treated as per whichever of the above
>> options we choose.
>>
>> My vote is that the text fixtures exported by a project live in that
>> project, which means either option 2 or 3. Option 2 might be the better
>> option. And it might be the better option for the Eclipse plugin as a
>> default behaviour, too.
>>
>> Short term fix is to move core's (and plugin's) test fixtures back into core
>> and plugin.
>
> I'd really rather not unless directed. It shouldn't be too difficult to do
> the right thing. At the very least, I'd like to separate core tests from core
> test fixtures. In pulling out the internalTesting project I had to detangle a
> few things and I'd rather not open the possibility up to do that again.
Sorry, I just mean back into the core and plugin projects, not necessarily into
the test source sets. I think keeping the core tests and the core test fixtures
separate is a very good idea.
--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com
