2012/11/7 Dawid Weiss <[email protected]>: > http://labs.carrotsearch.com/randomizedtesting.html > https://github.com/carrotsearch/randomizedtesting
I'll take a look at that. The only thing new to surefire in this patch is really dynamic allocation of tests to forked processes, since all the other stuff you mention is already supported. > > Some things to watch for, based on this experience: > > 1) Lots of hard to fix bugs will occur for users once you split > execution across forked JVMs if there are dependencies between suite > classes (pollution of static fields, left-over threads etc.). These > are hard to debug enough if you have a predictable suite assignment > (which can be done, see the runner's static assignment policy) but if > you have dynamic job stealing the order is pretty much a race > condition and the information which suites ran on which JVM (and in > which order) is crucial. Right; there is little chance of actually tracking what went wrong in the current logging regime; it would need to be beefed up with which vm and when things happened. > > 2) Didn't look into Andreas's code and don't know how he solved the > problem of communicating between JVMs. This is quite tricky if you > want to detect hung/crashed JVMs and capture the output they dump to > stdout/stderr descriptors directly (bypassing any System.* > replacements). We tail event files manually, this turned simple and > robust. This is basically unchanged in his patch; we capture all process output. > > 3) Most folks in Lucene were very attached to seeing plain console > output, even if concurrent JVMs are executing tests. I don't need to > mention this is again tricky and requires some coordination. Instead > of synchronizing slave VMs with the master (possibly delaying the > tests) we again went with tailing -- the forked VMs emit console > events much like they do anything else and once the master spots a > suite/ test was completed it pipes any output it produced to user's > console. There are many different scenarios here and all of them have > had both supporters and haters -- I'm not going to delve into it. This was also solved in surefire around version 2.6 or so and is unchanged in his patch; we also buffer until test completion. Output from static initializers/beforeclass when running concurrently within a single vm are the only glitches I am aware of in this logic. > > 4) Many test suites implicitly rely on the fact that they're executed > in isolation. This may result in unexpected failures as in two test > suites try to write to the same file in a current working directory, > for example. We opted for running each forked JVM in a separate CWD so > that they kind of get their own scratch space to work with. You could > agree that this is catering for badly designed tests but the reality > is many people will just have such tests in their code and won't know > why it's not a good idea to use cwd/ the same socket/ the same > resource across different suites. Yuck. Andreas's patch sets distinct environment variable within each forked vm, which is about as far as I think it's /nice/ to go. Separate cwd sounds very un-maven like; I wouldn't really expect maven users to have this problem ;) for all the other concurrent modes, there's tons of reasons why it won't work for any "random" code you try to run in parallel. I can live with that. We're not adding support for starting vmware images ;) Kristian --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
