I agree with Romain & your last couple sentences. I would prefer to expose faulty tests. Have you filed a JIRA with your diagnosis about the interaction? Standard caveat is that we are very close to being able to switch basic dev to gradle, with just some complex IT configs and release still using poms, so the issue might be slightly different.
The best case is that the build system isolates them from each other and wipes out the sandbox afterwards, though that can encourage tests to be sloppy about cleaning up. Kenn On Tue, Jan 30, 2018 at 8:38 AM, Romain Manni-Bucau <[email protected]> wrote: > Hi Daniel, > > As a quick fix it sounds good but doesnt it hide a leak or issue (in test > setup or in main code)? Long story short: using a random order can allow to > find bugs faster instead of hiding them and discover them randomly adding a > new test. > > That said, good point to have it configurable with a -D or -P and be able > to test quickly this flag. > > > Le 30 janv. 2018 17:33, "Daniel Kulp" <[email protected]> a écrit : > >> I spent a couple hours this morning trying to figure out why two of the >> SQL tests are failing on my machine, but not for Jenkins or for JB. Not >> knowing anything about the SQL stuff, it was very hard to debug and it >> wouldn’t fail within Eclipse or even if I ran that individual test from the >> command line with -Dtest= . Thus, a real pain… >> >> It turns out, there is an interaction problem between it and a test that >> is running before it on my machine, but on Jenkins and JB’s machine, the >> tests are run in a different order so the problem doesn’t surface. So >> here’s the question: >> >> Should the surefire configuration specify a “runOrder” so that the tests >> would run the same on all of our machines? By default, the runOrder is >> “filesystem” so depending on the order that the filesystem returns the test >> classes to surefire, the tests would run in different order. It looks >> like my APFS Mac returns them in a different order than JB’s Linux. But >> that also means if there is a Jenkins test failure or similar, I might not >> be able to reproduce it. (Or a Windows person or even a Linux user using >> a different fs than Jenkins) For most of the projects I use, we generally >> have “<runOrder>alphabetical</runOrder>” to make things completely >> predictable. That said, by making things non-deterministic, it can find >> issues like this where tests aren’t cleaning themselves up correctly. >> Could do a runOrder=hourly to flip back and forth between alphabetical and >> reverse-alphabetical. Predictable, but changes to detect issues. >> >> Thoughts? >> >> >> -- >> Daniel Kulp >> [email protected] - http://dankulp.com/blog >> Talend Community Coder - http://coders.talend.com >> >>
