[
https://issues.apache.org/jira/browse/LUCENE-4463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471090#comment-13471090
]
Dawid Weiss commented on LUCENE-4463:
-------------------------------------
I thought about it for a bit longer and exercised a few scenarios. The problem
is that I designed everything (and I mean everything) with two ideas in mind:
- every random element (be it a selection of components, shuffling of order or
whatever) is a derivative of a single "master" seed. This seed is picked by
<junit4> task and is then used to sort suites to be executed, pick parameters,
then is passed to suites to log messages, stack traces, etc.
- execution of a test suite (in the sense of a single class) is isolated from
anything else -- any other class running before or after. So you can provide
the same master seed for a single class and it should execute identically, even
if it's detached from the entire sequence of suites than ran during the full
test. The "seed decorators" that we currently use alter the master seed with a
hash of the test class's name to make it different for each class running under
the same master seed, but this is an independent operation -- whether something
ran before or after doesn't matter.
The idea of running the same suite many times with a _different_ master seed
each time conflicts with these assumptions because then every subsequent
execution of the same class will _not_ be a derivative of the master seed
anymore (and will most likely depend on how many classes executed before or
even be random).
Let me illustrate this on an example. Let's say the master seed is XXX; we use
this seed to pick file.encoding and for this seed it becomes UTF-8. If we now
pick a random master seed (say, YYY) for concrete class and it fails, it'll
report YYY back to the console. But if you ant -Dtests.seed=YYY then the
selection of file.encoding will be different because, ehm, it's not XXX
anymore. file.encoding has to be picked before the JVM is started so it cannot
be done from within the running test runner, etc.
This is just one of the problem scenarios, there are more but I hope you get
the picture.
A "clean" solution to the problem would be to make a loop inside ant, around
the contents of the test-macro (so that the entire sequence of picking the
master seed, picking parameters, spawning JVMs, etc. is repeated). This isn't
really going to make matters much faster because it'll fork new JVMs etc.
A "dirty" solution is to screw the above idealistic point of view and have a
seed decorator which affects the master seed before it is propagated to each
suite. This will cause all the headaches mentioned above PLUS you'll have to
get the failing seed directly from the failing test (stack trace or whatever
other message is printed) because it won't be the master seed JUnit4 greets you
with. Then you could indeed run as many concurrent instances of the same suite
with random seeds as you like (JVMs reused). This does sound like
super-advanced and convoluted piece of functionality for something that will be
probably used pretty frequently (which means lots of wtfs on the mailing list).
Don't know, really.
> Add support for running the same test method/class many times with different
> class seeds
> ----------------------------------------------------------------------------------------
>
> Key: LUCENE-4463
> URL: https://issues.apache.org/jira/browse/LUCENE-4463
> Project: Lucene - Core
> Issue Type: Wish
> Components: general/build
> Reporter: Robert Muir
> Assignee: Dawid Weiss
> Attachments: LUCENE-4463.patch
>
>
> I have a shell script for this, mike has a python script, its annoying :)
> I want to do something like this:
> ant beast -Dtestcase=XXXX -Dtestmethod=YYYY -Diterations=100
> I would be happy with a simple loop that just invokes 'test' somehow: getting
> a fresh new JVM to each iteration is desirable anyway (so you get fresh
> codecs, etc).
> the -Dtests.iters is not really useful for this because it does not allow
> -Dtestmethod and it does not give a fresh jvm.
> bonus points if it can use multiple jvms at the same time though :)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
