Hi Dawid, Thanks for the very good explanation! Indeed the main problem with tests.iters is the static initializers. Maybe put that explanation into the Wiki! I sometimes also need to remember it, so it should be documented.
One (only theoretical) way to solve the whole thing could be: Load the class(es) in a separate classloader for every repeated execution,... but of course this will very fast blow up your permgen (java 6, 7) or anything else we don't know about (java 8). In fact the separate classloader approach is not different from Mike's scripts, just that Mike's script creates a new classloader by forking a new JVM. In fact I don't think the separate classloader approach would be much faster, because the class clones will all have separate compilation paths in Hotspot, so Hotspot cannot share the same assembler code. So except the JVM startup time, you gain nothing. Just permgen issues :-) Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Dawid Weiss [mailto:dawid.we...@gmail.com] > Sent: Friday, August 08, 2014 3:10 PM > To: dev@lucene.apache.org > Subject: Re: Test iterations > > Hi Ryan, > > So. I discussed this a while ago, but here it comes again. Let me first clear > a > few things from what you said. > > > Only in the last month or so did I learn that -Dtests.iters doesn't really > "work". What I mean is in regards to randomization. > > This is not true. It works (as I will explain below). Try it, for example (the > annotation has the same effect as providing > -Dtests.iters=10): > > @Repeat(iterations = 10) > @Seed("0") > public class Test001 extends RandomizedTest { > @Test public void test() { > System.out.println(randomAsciiOfLength(10)); > } > } > > I fixed the initial seed to make it reproducible. This will print: > > nKtjLXhWQw > awHTHLIGAq > vEYgnxTkWv > mSAloRXtIV > iBhCJuZNzP > DHAIyqecSS > zaEoTAWAOa > CoraUrKuib > fKxUZnyQTx > beFtvsUTHc > > > Each iteration currently is *exactly* the same as far as randomization > >> (each iteration uses the same master seed). > > You can see above that it isn't true. Every iteration is different and uses > different randomness (and this randomness is "derived" from the (master, > iteration) pair so it is fully reproducible in each run). > > >> Why not create a different seed for each iteration when -Dtests.iters is > used? > > Let's talk about JUnit unit tests and how (any) runner should execute them. I > will demonstrate this on a simple class like this one (pseudo > code): > > class Foo { > @BeforeClass beforeClassHook() {} > > @Before beforeHook() {} > @Test test1() {} > @After afterHook() {} > > @AfterClass afterClassHook() {} > } > > There are a couple of "stages" to be executed. Simplifying a bit, it looks > like > this. > > 0. Prerequsite > > - class available, possible loaded and initialized > > 1. Setup: > > - extract test methods > > 2. Execution. > > - run class-before hooks (rules, @BeforeClass) > - for each test: > run before hooks (@Before, rules) > run the test itself > run after hooks (@After, rules) > - run class-after hooks (rules, @AfterClass) > > For the class above, the sequence of method calls would be: > > beforeClassHook() > > new() // constructor > beforeHook() > test1() > afterHook() > > afterClassHook() > > If you were to multiply tests execution manually, you would copy-paste the > test method giving it a different name: > > class Foo { > @BeforeClass beforeClassHook() {} > > @Before beforeHook() {} > @Test test1() {} > @Test test2() {} > @After afterHook() {} > > @AfterClass afterClassHook() {} > } > > which would result in a sequence of calls like this one: > > beforeClassHook() > > new() // constructor > beforeHook() > test1() > afterHook() > > new() // constructor (new instance) > beforeHook() > test2() > afterHook() > > afterClassHook() > > So, first of all, note that duplicating tests is *not* equivalent to just > looping > around method body. Each execution should be run on a new instance and > wrapped with setup and teardown hooks, otherwise it's not really an isolated > JUnit test anymore (and it would be against JUnit informal execution flow). > > This is, in short, what -Dtests.iters does (and what @Repeat does) -- it > replicates every test, making sure they have unique names (IDEs get > confused if they don't) and trying to work around other issues I won't discuss > here. It does work. The reason you believe it doesn't work is because most > of the stuff in LuceneTestCase is initialized at *static* class level, which > by > definition is executed only once, regardless of the number of tests in a > class. > Let's modify our initial example a > bit: > > @Repeat(iterations = 10) > @Seed("0") > public class Test002 extends RandomizedTest { > static String s; > > @BeforeClass > public static void beforeClass() { > s = randomAsciiOfLength(10); > } > > @Test public void test() { > System.out.println(s); > } > } > > If you run this, you'll see: > > SXVNjhPdQD > SXVNjhPdQD > SXVNjhPdQD > SXVNjhPdQD > SXVNjhPdQD > SXVNjhPdQD > SXVNjhPdQD > SXVNjhPdQD > SXVNjhPdQD > SXVNjhPdQD > > This works as expected because beforeClass() is invoked once (even if every > test has a different randomness available to it). LuceneTestCase does it for > performance reasons (that static initialization is fairly costly). This ends > the > JUnit part of the story. > > But wait, there is more. If you take a look above at how JUnit runners should > work they load the class (or in fact are given an initialized > class) before they can do anything. So if there are static class initializers > (static > { field = foo(); }) then these may get executed well before the runner has > any chance to initialize its own stuff -- that's why you *have* to use > @BeforeClass methods if you want to use RandomizedTest's randomness; > doing > this: > > @Repeat(iterations = 10) > @Seed("0") > public class Test003 extends RandomizedTest { > static final String s; > static { > s = randomAsciiOfLength(10); > } > > @Test public void test() { > System.out.println(s); > } > } > > will result in an initialization exception complaining about missing random > context: > > java.lang.IllegalStateException: No context information for thread: > Thread[id=11, name=SUITE-Test003-seed#[0], state=RUNNABLE, > group=TGRP-Test003]. Is this thread running under a class > com.carrotsearch.randomizedtesting.RandomizedRunner runner context? > Add @RunWith(class > com.carrotsearch.randomizedtesting.RandomizedRunner.class) to your test > class. Make sure your code accesses random contexts within @BeforeClass > and @AfterClass boundary (for example, static test class initializers are not > permitted to access random contexts). > > Finally, all the above probably begs the question why can't the runner just > wrap every iteration one level up the ladder -- re-execute static hooks, etc. > > The answer is that you are only considering the level of a single class and > the > master seed is in fact used for other things, most notably for ordering test > classes for predictable execution within one forked JVM (this is done in > JUnit4-Ant task). If you re-executed one class's static hooks with a different > seed for different iterations how would they report the "real master" seed > back for re-running the entire suite? Should it be a pair master- > seed:iteration? Only the derived class seed? Or maybe we should have two > seeds -- one for ordering classes and one for the actual class execution? > > It gets really messy if you get down to details like this, not to mention > missing > support from IDEs for running multiple tests of the same class (it's already > pretty complicated to make it work in Eclipse, IntelliJ Idea, etc. when > multiple > identical methods are executed). > > Finally: > > >> different people have their own "beasting" scripts that run the test > >> essentially N times from a shell to force different seeds in each > >> iteration. > > This is true. ButI don't think Mike, for example, will resign from his scripts > even if "real" class-level tests.iters would be implemented -- Mike's script > runs tests across multiple machines via SSH; I won't have the time for such > distributed extensions in any near future. > > If you want to try to implement reiteration at the 'static context' > feel free to give it a shot though; I would certainly be interested in how you > approach the problems I mentioned above. A good place to start would be to > modify JUnit4-ant (Ant task), around here: > > https://github.com/carrotsearch/randomizedtesting/blob/master/junit4- > ant/src/main/java/com/carrotsearch/ant/tasks/junit4/JUnit4.java#L887 > > if you somehow associated a class with its master seed (and duplicated > classes as well), then you could fork off multiple class executions with > different seed (you'd still have to modify the forked subprocess to accept > class name and master seed; currently only one master seed is passed at the > start of each JVM. > > Or you could do what I mentioned above -- separate the seed used for class > ordering from the one used for executing every class (every iteration of a > class). > > Or maybe it'd be easier to modify the runner itself (then duplication would > work from IDE level)... but then you'll hit the IDE issues and constraints... > I > really don't know which way is better (or worse). > That's part of the challenge I guess. :) > > I'll try to look at the code again in the next few days and will give you some > feedback. I can't log to Jira for some reason, but I'll lookup the issue > number > for this, it's been there for a good while. > > Dawid > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional > commands, e-mail: dev-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
