On Thu, 2016-12-08 at 10:46 +0100, Julian Reschke wrote: > On 2016-12-07 12:21, Michael Dürig wrote: > > > > In a F2F discussion we agreed to pay more attention to test > > failures on > > the Apache Jenkins instance and make an effort to turn it green: > > > > - tests that are constantly failing and also fail on a local > > checkout > > should be marked @Ignored along with an issue reference and an bug > > report in Jira. > > > > - tests that only fail on Jenkins should be marked as such through > > the > > CIHelpers facility > > > > In the meanwhile I tried to configure the Jenkins Jira plugin so it > > would automatically file Jira issues for failures. This didn't seem > > to > > successful as the last failing build didn't show up in the Oak > > Jira. > > There is no indications of anything going wrong in the build logs > > neither though. Anyone who wants to take this up, be my guest. > > > > Michael > > +1 > > FWIW; we have way too many tests that happen to fail "sometimes". > The > following just happened twice to me when running integration tests: > > Failed tests: > externalAddOffline(org.apache.jackrabbit.oak.plugins.blob.datastore.B > lobIdTrackerTest): > expected:<1006> but was:<1001>
There is a plug-in named 'Test results analyser' [1] installed on the ASF Jenkins instance which exposes test execution results over a number of jobs. We have it enabled for Sling and you can see some of the flaky tests 'in action' at https://builds.apache.org/view/S-Z/view/Sling-Dashboard/job/sling-ins taller-it-1.8/test_results_analyzer/ You might want to enable it to track how flaky tests really are. Robert [1]: https://wiki.jenkins-ci.org/display/JENKINS/Test+Results+Analyzer+ Plugin
