On 27/08/2011, at 12:10 AM, Adam Murdoch wrote: > > On 26/08/2011, at 6:07 PM, Luke Daley wrote: > >> >> On 26/08/2011, at 6:45 AM, Adam Murdoch wrote: >> >>> Hi, >>> >>> I'm not sure I like moving (some of) core/src/test out into a separate >>> project. As a general pattern, I think I would much rather that the test >>> fixtures exported by a project live in that project, rather than in a >>> separate project, and are modelled as a separate publication. >>> >>> That is, right now, we have the following dependencies: >>> >>> core test -> internalTesting main -> core main >>> >>> I'd much rather: >>> >>> core test -> core testFixtures -> core main. >> >> core-test is going to have a compile dependency on core-main as well though, >> so it's not quite that linear (unsure if that's relevant here). >> >>> where 'core testFixtures' are the core-specific test fixtures, such as >>> classes like HelperUtil, AbstractTaskTest, AbstractTaskSpec. This doesn't >>> include int testing fixtures such as GradleExecutor, and probably doesn't >>> include general things such as TemporaryFolder, TestFile, and so on. >> >> Makes sense. If you are compiling against core, then you'll likely want to >> compile against coreTestFixtures. >> >> However, at some point we'll probably want to publish a core-fixtures jar of >> public utilities for this and we may want to actually have tests for these >> test fixtures. This could be the same for any plugin, e.g. we might want to >> publish a maven-fixtures jar for unit test fixtures for developers who >> integrate with/use types from the maven plugin(s). >> >> The same could be said for integration tests. There may be a >> maven-integ-fixtures that provides an in memory http server for setting up >> repos for integration tests. >> >> I think this _potentially_ leaves us with the following groups of source for >> a project >> >> main >> test >> testFixtures >> integTest >> integTestFixtures >> >> If we do want to have public test fixtures that might mean another four >> (names aside): >> >> publicTestFixtures >> publicTestFixturesTest >> publicIntegTestFixtures >> publicIntegTestFixturesTest > > I think there's no question all these categories of code will exist. However, > I'm not sure we necessarily want to split each category out into its own > source set. We do have some other dimensions we can use to categorise code, > and we could use a mix of these: > > * Split into projects > * Split into source sets > * Split into package hierarchies > * probably some others too > > I'm not sure what a good scheme would be, yet. I think we could simplify this > a bit for now, let it soak for a while and see what we come up with. > > I'd suggest something like this, for now: > > * main source set -> any public code, regardless of type. > * testFixtures source set -> internal test fixtures, regardless of type. > * test source set -> unit tests for any code used outside the project (ie > main and test fixtures) > * integTest source set -> int tests for any code used outside the project
That's what I did yesterday and it seems to work well. > One benefit of splitting out the tests and test fixtures is that a change to, > say, a unit test in core won't go invalidating the unit and integration tests > in all the other projects. > > Another benefit of splitting out test fixtures is that we can reuse these > internal test fixtures in our other Gradleware work. So, 'internal' above > means the same thing as 'internal' for APIs: it is primarily intended to be > used inside Gradle, but you can use it if you like. Just keep in mind it's > unstable and may change at any time. > > We'd keep the internalTesting project, it would contain only testFixtures. Putting the internalTesting main classes into testFixtures doesn't feel right to me. They should be in main as that's the main point of this project. That's the way this is currently setup. This makes more sense conceptually to me. > Not sure what to do with the integ test fixtures. The question is whether > integration testing is conceptually part of core or not. It feels to me like > it is a separate enough concern to warrant its own project. Given this, we > wouldn't necessarily need an integTestFixtures source set, as the integTest > project would have: > > * main -> public integ test fixtures > * testFixtures -> internal integ test fixtures > * test -> unit tests for the fixtures > * integTest -> integ tests for the fixtures I created internalIntegTesting yesterday that contains GradleDistributionExecuter and friends, but the source lives in main for the same argument I gave above for internalTesting. Let's park the discussion about public versions of this stuff for the time being until that firms up a little. >> This is starting to look like too much, but all of these (apart from main) >> would be optional. I wouldn't expect many plugins to expose public utilities >> for integration testing but it could definitely happen (and should in some >> cases). >> >> Also, I'd rather have this source set explosion and very carefully control >> what's public than try and jam them together (but I guess having so many >> projects may be burdensome in the IDE). >> >>> One problem - apart from the conceptual awkwardness - with having a >>> project's test fixtures in a separate project is that it blows our 'must be >>> able to import into eclipse' story right out of the water, as the eclipse >>> plugin will map these to: >>> >>> core -> internalTesting -> core >>> >>> which, of course, eclipse can't handle. >> >> If we move the integration testing harness out into its own project (as I >> think you said we should), won't we have the same problem? > > We will. This is why my three options below all had a project's integTest > source set mapped to a separate eclipse project. > > I think, if we tried, we could end up with something like: > > compile: > integTest main/fixtures > -> toolingApi main > -> wrapper main > (currently toolingAPI drags in core, but shouldn't) > > int test compile (using scala as an example): > scala int test -> > -> scala main > -> plugins main > -> core main > -> integTest main/fixtures > > int test runtime (forking mode): > scala int test > -> integTest main/fixtures > -> toolingApi main > -> wrapper main > -> gradle dist (which happens to contain scala main) > > int test runtime (embedded mode): > scala int test > -> scala main > -> plugins main > -> core main > -> integTest main/fixtures > -> toolingAPI main > -> wrapper main > -> launcher main > -> core main > -> coreImpl main > -> ui main > > The basic idea is that integTest fixtures drag in only the tooling API and > launcher at runtime. These would then make use of whatever plugins they can > find on the classpath at runtime. Currently, the integ test fixtures drag in > all projects at runtime, which means a huge cycle. > > A project would automatically pick up any plugins which it has a > compile/runtime dependency on, and the project can add in more by adding an > integTest compile/runtime dependency. > > A benefit of this approach is that a change to, say, the announce plugin, > does not invalidate the integration tests for, say, c++. And that to run the > int tests for the ide plugin, I don't need to build the docs. I want to do something like this soon as it would be a productivity improvement for me if I needed to build less to run integration tests. Basically, I want to rework the intTestImage stuff to build an image per integration test run (i.e. each project) so only what is strictly needed is built. I have some ideas on how to do this. > I think this works, except if any of integTest's runtime dependencies need to > have integration tests (eg core, tooling API, or ui). Hmm, that's an interesting one. I can see that core and tooling api will. > The (perhaps simpler) alternative, is that the integTest source set of every > project is mapped to a separate ide project. Then we can have whatever > dependencies we like (including the above, of course). I think this makes sense. >>> Another issue (for me) is that idea treats the test fixtures as production >>> code, not test code. This breaks things such as search for usages. >> >> So IDEA knows that the test source sets etc. are not “production”? >> >> Does this also mean that for the “public” test fixtures we'd need to have >> them as separate projects so IDEA knows they are production code? > > They'd just need to be in a source directory that we can mark as a production > source dir. This might mean that they live in the main source set of the same > project, or in a separate source set, or in a separate project. > >> >>> I think there are some better options: >>> >>> 1. We push the separate projects option even further, and bust up every >>> project into 4 separate projects: main, testFixtures, tests, and integTests. >>> >>> 2. Test fixtures live in the owning project, and we map each source set to >>> a separate eclipse project, so we end up with -main, -test, and -integTests >>> eclipse projects. Possibly combine this with adding a testFixtures source >>> set as well. >> >> I don't understand the last sentence. Does that just mean a -testFixtures >> project (just like -integTests) or something else? >> >> This seems like the winner to me. >> >>> 3. Test fixtures live in the owning project, and we map the integTest >>> source set of each project to a separate eclipse project. Possibly combine >>> this with adding a testFixtures source set as well. >>> >>> What is currently 'internalTesting' would split into 3 pieces: the general >>> purpose test fixtures, the core project's test fixtures, and the >>> integration testing fixtures. These would be treated as per whichever of >>> the above options we choose. >>> >>> My vote is that the text fixtures exported by a project live in that >>> project, which means either option 2 or 3. Option 2 might be the better >>> option. And it might be the better option for the Eclipse plugin as a >>> default behaviour, too. >>> >>> Short term fix is to move core's (and plugin's) test fixtures back into >>> core and plugin. >> >> I'd really rather not unless directed. It shouldn't be too difficult to do >> the right thing. At the very least, I'd like to separate core tests from >> core test fixtures. In pulling out the internalTesting project I had to >> detangle a few things and I'd rather not open the possibility up to do that >> again. > > Sorry, I just mean back into the core and plugin projects, not necessarily > into the test source sets. I think keeping the core tests and the core test > fixtures separate is a very good idea.
