> some of the awesomest stuff I've seen in a long while, I simply have I implemented such stuff for Apache Lucene/ Solr a while back, it's been running for quite a while now. It's not technically Maven (it's an Ant task) but a Maven plugin wrapper for it is also provided and does an equivalent thing -- load-balancing of test suites across workers based on previous execution statistics, job-stealing to minimize the execution time when times are unknown (or vary across runs).
http://labs.carrotsearch.com/randomizedtesting.html https://github.com/carrotsearch/randomizedtesting Some things to watch for, based on this experience: 1) Lots of hard to fix bugs will occur for users once you split execution across forked JVMs if there are dependencies between suite classes (pollution of static fields, left-over threads etc.). These are hard to debug enough if you have a predictable suite assignment (which can be done, see the runner's static assignment policy) but if you have dynamic job stealing the order is pretty much a race condition and the information which suites ran on which JVM (and in which order) is crucial. 2) Didn't look into Andreas's code and don't know how he solved the problem of communicating between JVMs. This is quite tricky if you want to detect hung/crashed JVMs and capture the output they dump to stdout/stderr descriptors directly (bypassing any System.* replacements). We tail event files manually, this turned simple and robust. 3) Most folks in Lucene were very attached to seeing plain console output, even if concurrent JVMs are executing tests. I don't need to mention this is again tricky and requires some coordination. Instead of synchronizing slave VMs with the master (possibly delaying the tests) we again went with tailing -- the forked VMs emit console events much like they do anything else and once the master spots a suite/ test was completed it pipes any output it produced to user's console. There are many different scenarios here and all of them have had both supporters and haters -- I'm not going to delve into it. 4) Many test suites implicitly rely on the fact that they're executed in isolation. This may result in unexpected failures as in two test suites try to write to the same file in a current working directory, for example. We opted for running each forked JVM in a separate CWD so that they kind of get their own scratch space to work with. You could agree that this is catering for badly designed tests but the reality is many people will just have such tests in their code and won't know why it's not a good idea to use cwd/ the same socket/ the same resource across different suites. Don't know if any of the code in that github account will be of use for you -- RandomizedRunner is designed with some specific goals in mind -- but I do have some experience in the matter of running JUnit tests concurrently and if this can be helpful ping me. Dawid --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
