Okay, one of my recent commits makes the tests deterministic. When we have them all passing, we can see if they pass for all seeds.
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Itamar Syn-Hershko Sent: Monday, September 8, 2014 5:06 PM To: [email protected] Subject: Re: Dealing with randomness in unit tests Yes, this is a known problem. Lucene's test randomization was contributed by CarrotSearch and specifically Dawid Weiss. His talk on the subject is a must-watch really http://vimeo.com/32087114 I've discussed this with him and it appears compiling CarrotSearch's tests randomization library with IKVM is the way to go. I have received some integration classes they use internally to connect this with NUnit. I have yet to do anything about it. If we can disable the randomness for now by just using a static seed without changing any code, that would be great. This way we could keep productive and introduce predictive randomness using carrotsearch's excellent codebase when we get the time to do so. Does that make sense? -- Itamar Syn-Hershko http://code972.com | @synhershko <https://twitter.com/synhershko> Freelance Developer & Consultant Author of RavenDB in Action <http://manning.com/synhershko/> On Tue, Sep 9, 2014 at 2:59 AM, Prad Nelluru <[email protected]> wrote: > Hey all, > > The Java Lucene code uses a test randomization framework. The > framework's random number methods use a new seed value each time the tests > are run. > Whenever there's a test failure, the seed is displayed so the > conditions for the failure can be replayed. As there is no equivalent > randomization framework in C#, we currently just use the .net Random > class whenever a random generator is required. Crucially, we don't > provide a seed or even record it. As the seeds can't be known, we have > no way of reproducing test failures. While I haven't seen an > intermittent failure yet, I've heard that Russell ran into them occasionally. > > We can either use a hardcoded fixed seed, or we can store a random > seed to show when a test fails. In any case, we should remove the > initializations of Random all over the place (replacing them with > LuceneTestCase.Random()). > > What do you all think? > > -Prad >
