On Sun, May 22, 2016 at 10:12 AM, Stack <[email protected]> wrote: > On Fri, May 20, 2016 at 3:43 PM, Todd Lipcon <[email protected]> wrote: > > > On Fri, May 20, 2016 at 1:17 PM, Matteo Bertozzi < > [email protected]> > > wrote: > > > > > any suggestion on how to make people aware of the tests being flaky? > > > > > > > You guys might consider doing something like what we do for Apache Kudu > > (incubating): > > > > http://dist-test.cloudera.org:8080/ has a dashboard (driven from our > > flaky-tests job) which shows the percent flakiness of each test, as well > as > > a breakdown of pass/fail rates by revision. We don't automatically email > > these to the list or anything, currently, but would be pretty easy to set > > up a cron job to do so. > > > > The dashboard is very helpful for prioritizing the de-flaking of the > worst > > offenders, and also useful to quickly drill down and grab failure logs > from > > the flaky tests themselves. > > > > > Would you suggest copy/paste of your current setup (a python daemon and a > db instance IIRC)? >
Sure, you're welcome (and encouraged) to borrow/steal it. If you make some improvements, please let us know, though, so we can merge them back to our copy The code for the server is here: https://github.com/apache/incubator-kudu/blob/master/build-support/test_result_server.py The only bit that's kudu-specific is the 'parse_test_failure' module - it has some heuristics to try to pull out the failure error message from our tests, but could easily be left out. -Todd -- Todd Lipcon Software Engineer, Cloudera
