So to summarize: -Dtests.iters generate different seeds for each suite/run. It's good if you have a random test and want to test its (or the code's) robustness.
-Dtests.dups uses the same seed for every suite, but can run in parallel JVMs. Not sure what's the use case though ... what's the point of running w/ same seed multiple times? Maybe only to test concurrency, of a test that is not highly concurrent itself? Looks like if we could enjoy both worlds, generate a different seed for every invocation and run them in parallel, that would be great. But I'm sure Dawid has thought about it, and there's some JUnit limitation that does not allow it :). Shai On Sun, Dec 9, 2012 at 1:13 PM, Shai Erera <ser...@gmail.com> wrote: > Yeah you explained it to me (at least once !) > > > Filters are applied outside of the scope of >> the runner >> > > That's the part I keep forgetting ! :) > > > tests.dups returns the whole suite with the same random seed, so every run >> is 100% identical >> > > Ah, that's important. I just verified that and you're right. Ran a test w/ > -Dtests.iters=2 and -Dtests.dups=5, every invocation used same seeds. > We should document that in test-help ! > > So now I'm even more confused. What is tests.dups good for then? Just > running the same suite multiple times, but not changing the seed? That can > be achieved w/ tests.iters and tests.seed .. > > Dawid, can tests.dups use different seeds? Then it'd be really useful > cause it can run these iters in parallel. > > And then we can document that tests.iters + glob is what you should call > from eclipse ... > > Shai > > > On Sun, Dec 9, 2012 at 12:56 PM, Dawid Weiss <dawid.we...@cs.put.poznan.pl > > wrote: > >> I think I wrote about it a few times -- in short: what you ask for is >> not possible with the current JUnit architecture and Eclipse runner's >> implementation. >> >> The problem is that every test method has a "description" object that >> needs to be a unique combination of 'suite name' and 'method name' >> (this is the case with methods in Java so it's not a problem for >> regular tests). If you want to re-run the same test a few times in >> JUnit you need to differentiate every run; otherwise method names >> won't be unique and this confuses every IDE and reporting tool out >> there. Randomized testing concatenates non-unique descriptions with >> the seed and with a sequential number to enforce this uniqueness. >> >> Then there are filters. Filters are applied outside of the scope of >> the runner, so when you filter a test method in Eclipse it simply >> applies a pattern over a description. So a description "enhanced" with >> the seed and method number has no chance of being matched and is >> filtered out -- this is why you get an empty set of tests after trying >> to run with tests.iters and a filter. When you specify -Dtests.method >> then the filtering is applied at the runner level (as opposed to >> Eclipse's filter). >> >> I use Eclipse as well and believe me, I tried very hard to make it >> work but there's just no way to do it. >> >> As for tests.dups -- this will work under ant only because what it >> does is it duplicates entire suites, not methods. This again is not >> really JUnit compatible... >> >> Dawid >> >> >> >> On Sun, Dec 9, 2012 at 10:18 AM, Shai Erera <ser...@gmail.com> wrote: >> > Hi >> > >> > Can the randomizing test framework add the glob implicitly, if a >> > -Dtests.method is defined together with -Dtests.iters? >> > I don't mind adding the glob, but it'd be much nicer and easier if the >> > framework did it. Like, it could just check that the specific test >> method >> > invocation startsWith tests.method? >> > >> > The reason I'm asking is because when I run a single test method from >> > eclipse and add -Dtests.iters, it doesn't work. >> > And I always forget that you cannot run them like that. You have to run >> the >> > entire test-class, and add the tests.method filter. >> > But then I forget to remove the filter when I actually want to test the >> > entire class ... or if I do remember that, I need to create >> > several run configurations with names ... >> > >> > This is just a convenience. If JUnit doesn't allow that, I'll live >> without >> > it. >> > >> > Shai >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >> >
