On Fri, May 4, 2012 at 6:29 PM, Chris Hostetter <hossman_luc...@fucit.org> wrote: > > Dawid: > > With the new test runner you created, would it be possible to setup an > annotation that we could use instead to indicate that a test should in fact > be run, and if it fails, include the failure info in the build report, but > do not fail the build? > > I'm thinking in particular about some of the test that multi-threaded tests > that are currently marked @Ignore or @AwaitsFix because they sporadically > fail on jenkins in our jail -- but that people haven't been able to > reproduce consistently on local dev machines (or that some people have been > able to reproduce, but not the people who understand the tests/code well > enough to try and fix) > > As it stands right now, if somone wants to try and fix a complicated test > that's disabled, they have to make a guess at the fix, un-@Ignore, then > watch the next few/several builds patiently to see if / how-often it fails, > then commit the @Ignore back, and repeat. > > If we could leave these tests running on every build, then we could at least > monitor the relative frequency of the failures -- ie: "last week testFoo > failed in 10% of the builds, this week it fails in every build, so somebody > definiteily broke something" or "last week testFoor failed in 10% of the > builds, and after my attempted hardening it only fails in 5% of the builds > so i may be on to something." > > what do folks think?
+1 Something like this is definitely needed. Some of the Solr tests that spin up multiple JVMs are particularly tough to get 100% bullet-proof on all platforms (esp this freebsd jail) and there is a lot of information in tests that occasionally fail (esp if said tests may be the *only* tests we have for certain functionalities). -Yonik --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
