Hello everyone, Hope (some of) you had a good break.
I've placed a draft PR pushing module system compatibility changes a bit further, the PR is here: https://github.com/apache/lucene/pull/571 I essentially got to the point of making tests in most subprojects run with dependencies (cross-project and external) on module-path. This already revealed a few issues where the module system turns out to behave differently (resource lookups) and I expect more of these to appear in the future. It's a bit like two different worlds. I had to make a few changes to the codebase and the reasons for these changes will also apply in the future. * I had to convert spatial test fixtures to a subproject. Gradle plugins that add or modify source sets and their paths will be a problem. They are basically an orthogonal dimension that complicates (or even makes it impossible) to assemble the module path correctly. I gave up on trying to make the test fixtures plugin working. * There are very few patterns and best practices on how to work with projects that have to run in both the classpath and modular world. A good writeup on how one can organize tests, for example, was written by Christian Stein (mentioned by Uwe previously) - https://sormuras.github.io/blog/2018-09-11-testing-in-the-modular-world.html However, I found the black-box automatisation (in both Maven and gradle) only gets you to a certain point and then stops working - you have to rewrite most of the stuff yourself anyway. All the different tasks that use classpath (javac, javadoc, ecj, tests) will require the same or, in many cases, different treatments for providing arguments and none of the built-in transparent tweaks work across tools, IDEs, etc. It quickly gets into a complicated mess of customized hacks. So, what works in PR 571 and what doesn't? The tests mostly work from the classpath, with project dependencies on module-path if they're modular dependencies. This is best explained on an example: > gradlew -p lucene\analysis\morfologik -Pbuild.debug.paths=true test ... Modular extension, runtime paths, mode=DEPENDENCIES_ON_MODULE_PATH: Module path: ...\randomizedtesting-runner-2.7.6.jar ...\junit-4.13.1.jar ...\morfologik-fsa-2.1.8.jar ...\morfologik-polish-2.1.8.jar ...\morfologik-stemming-2.1.8.jar ...\hamcrest-2.2.jar ...\morfologik-ukrainian-search-4.9.1.jar ...\lucene\analysis\common\build\libs\lucene-analysis-common-10.0.0-SNAPSHOT.jar ...\lucene\codecs\build\libs\lucene-codecs-10.0.0-SNAPSHOT.jar ...\lucene\core\build\libs\lucene-core-10.0.0-SNAPSHOT.jar ...\lucene\test-framework\build\libs\lucene-test-framework-10.0.0-SNAPSHOT.jar Class path: ...\lucene\analysis\morfologik\build\classes\java\main ...\lucene\analysis\morfologik\build\classes\java\test ...\lucene\analysis\morfologik\build\resources\main ...\analysis\morfologik\build\resources\test I've removed some extra leading paths above but you can see that the module path includes external dependencies (both named modules and automatic modules) and cross-project dependencies (including the test framework). The split of classes and resources (a convention introduced by Maven) and split-packages between tests and main classes of the project are the reason why we have the main source set's and test source set's artifacts on classpath: ...\lucene\analysis\morfologik\build\classes\java\main ...\lucene\analysis\morfologik\build\classes\java\test ...\lucene\analysis\morfologik\build\resources\main ...\lucene\analysis\morfologik\build\resources\test While there are technically ways of combining these folders into a module I found it too magical, at least for now. So PR 571 does not attempt to solve this. Fully modular tests are possible but you have to create a different subproject (with an extension '.tests' and then reference the tested module from there. Obviously the test module name and packages used have to be unique then. This is demonstrated on two subprojects - core.tests and morfologik.tests. For example, compare the printed paths for this: > gradlew -p lucene\core.tests -Pbuild.debug.paths=true test Modular extension, runtime paths, source set=test (module), mode=DEPENDENCIES_ON_MODULE_PATH: Module path: ...\junit-4.13.1.jar ...\hamcrest-2.2.jar ...\lucene\core.tests\build\libs\lucene-core.tests-10.0.0-SNAPSHOT-test.jar ...\lucene\core.tests\build\libs\lucene-core.tests-10.0.0-SNAPSHOT.jar ...\lucene\core\build\libs\lucene-core-10.0.0-SNAPSHOT.jar Class path: [empty] As can be seen, everything ended up on module-path. The background of how this works is pretty ugly (to me) but in short terms, the source set outputs are converted to JARs - the main source set would correspond to lucene-core.tests-10.0.0-SNAPSHOT.jar and the test source set is in lucene-core.tests-10.0.0-SNAPSHOT-test.jar. It does work and it works in full module-mode. What's remaining? Well, exploration. I've had some prototype work that had module descriptors both in the main source set and in the test source set - couldn't get it to work with IntelliJ (confused the heck out of it). I also have an idea how module patching could work to allow non-modular tests to work with a modular main source set... but I didn't have time to try it yet and I have a strong suspicion it will break IDEs again, so leaving it for now. Finally, there are problems of the chicken-end-egg nature - the following subprojects are explicitly using full classpath mode for compilation/tests (modules.java): ":lucene:core", ":lucene:codecs", ":lucene:spatial3d", ":lucene:spatial-extras", ":lucene:test-framework", these are there because of cyclic dependencies. For example, the tests of the test framework are on classpath (including the test framework itself) but the core (and test secrets accessors) are on module path - this causes runtime module layer exceptions that test secrets cannot be accessed from an unnamed module. Spatial modules use a split-package dependency. If anybody is willing to jump in, the PR contains the code. It's not easy - it's touching quite a lot of gradle APIs - but it's fairly self-contained in modules.java. Feel free to hack around and suggest improvements, I'm slowly running out of ideas if it can be made any easier/ better. Dawid --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
