To expand on what Robert says, many other things in our test framework are randomized. e.g. PCollection elements are shuffled randomly, bundle sizes are determined randomly, etc. All of this should be repeatable if there's a failure. The test should print the seed used to generate the random numbers, and you should be able to pass that seed back into the run to recreate those exact conditions.
On Tue, Jan 30, 2018 at 10:27 AM, Robert Bradshaw <[email protected]> wrote: > Agreed, any leakage of state between tests is a bug, and giving things > a deterministic order just hides these bugs. I'd be in favor of > enforcing random ordering (with a published seed for reproduciblity of > course). > > On Tue, Jan 30, 2018 at 9:21 AM, Lukasz Cwik <[email protected]> wrote: > > The order should be random to ferret out issues but the test order seed > > should be printed and configurable so it allows replaying a test run > because > > you can specify the order in which it should execute. > > > > I don't like having a strict order since it hides poorly written tests > and > > people have a tendency to just work around the poorly written test > instead > > of fixing it. > > > > On Tue, Jan 30, 2018 at 9:13 AM, Kenneth Knowles <[email protected]> wrote: > >> > >> What was the problem in this case? > >> > >> On Tue, Jan 30, 2018 at 9:12 AM, Romain Manni-Bucau > >> <[email protected]> wrote: > >>> > >>> What I was used to do is to capture the output when I identified some > of > >>> these cases. Once it is reproduced I grep the "Running" lines from > surefire. > >>> This gives me a reproducible order. Then with a kind of dichotomy you > can > >>> find the "previous" test making your test failing and you can > configure this > >>> sequence in idea. > >>> > >>> Not perfect but better than hiding the issue probably. > >>> > >>> Also running "clean" enforces inodes to change and increase the > >>> probability to reproduce it on linux. > >>> > >>> > >>> Romain Manni-Bucau > >>> @rmannibucau | Blog | Old Blog | Github | LinkedIn > >>> > >>> 2018-01-30 18:03 GMT+01:00 Daniel Kulp <[email protected]>: > >>>> > >>>> The biggest problem with random is that if a test fails due to an > >>>> interaction, you have no way to reproduce it. You could re-run with > random > >>>> 10 times and it might not fail again. Thus, what good did it do to > even > >>>> flag the failure? At least with alphabetical and reverse > alphabetical, if a > >>>> tests fails, you can rerun and actually have a chance to diagnose the > >>>> failure. A test that randomly fails once out of every 20 times it > runs > >>>> tends to get @Ignored, not fixed. I’ve seen that way too often. :( > >>>> > >>>> Dan > >>>> > >>>> > >>>> > On Jan 30, 2018, at 11:38 AM, Romain Manni-Bucau > >>>> > <[email protected]> wrote: > >>>> > > >>>> > Hi Daniel, > >>>> > > >>>> > As a quick fix it sounds good but doesnt it hide a leak or issue (in > >>>> > test setup or in main code)? Long story short: using a random order > can > >>>> > allow to find bugs faster instead of hiding them and discover them > randomly > >>>> > adding a new test. > >>>> > > >>>> > That said, good point to have it configurable with a -D or -P and be > >>>> > able to test quickly this flag. > >>>> > > >>>> > > >>>> > Le 30 janv. 2018 17:33, "Daniel Kulp" <[email protected]> a écrit : > >>>> > I spent a couple hours this morning trying to figure out why two of > >>>> > the SQL tests are failing on my machine, but not for Jenkins or for > JB. > >>>> > Not knowing anything about the SQL stuff, it was very hard to debug > and it > >>>> > wouldn’t fail within Eclipse or even if I ran that individual test > from the > >>>> > command line with -Dtest= . Thus, a real pain… > >>>> > > >>>> > It turns out, there is an interaction problem between it and a test > >>>> > that is running before it on my machine, but on Jenkins and JB’s > machine, > >>>> > the tests are run in a different order so the problem doesn’t > surface. So > >>>> > here’s the question: > >>>> > > >>>> > Should the surefire configuration specify a “runOrder” so that the > >>>> > tests would run the same on all of our machines? By default, the > runOrder > >>>> > is “filesystem” so depending on the order that the filesystem > returns the > >>>> > test classes to surefire, the tests would run in different order. > It looks > >>>> > like my APFS Mac returns them in a different order than JB’s > Linux. But > >>>> > that also means if there is a Jenkins test failure or similar, I > might not > >>>> > be able to reproduce it. (Or a Windows person or even a Linux > user using a > >>>> > different fs than Jenkins) For most of the projects I use, we > generally > >>>> > have “<runOrder>alphabetical</runOrder>” to make things completely > >>>> > predictable. That said, by making things non-deterministic, it > can find > >>>> > issues like this where tests aren’t cleaning themselves up > correctly. > >>>> > Could do a runOrder=hourly to flip back and forth between > alphabetical and > >>>> > reverse-alphabetical. Predictable, but changes to detect issues. > >>>> > > >>>> > Thoughts? > >>>> > > >>>> > > >>>> > -- > >>>> > Daniel Kulp > >>>> > [email protected] - http://dankulp.com/blog > >>>> > Talend Community Coder - http://coders.talend.com > >>>> > > >>>> > >>>> -- > >>>> Daniel Kulp > >>>> [email protected] - http://dankulp.com/blog > >>>> Talend Community Coder - http://coders.talend.com > >>>> > >>> > >> > > >
