Amazing idea! In my company we have developed a RunConfiguration to do exactly that and implemented a plugin to do the same thing (but haven't opened it because it become a very specific implementation)
Thumbs up for the initiative! Em 01/08/2014 21:44, "Qingzhou Luo" <[email protected]> escreveu: > Hi, > > I'd like to host our new plug-in. > > Plugin name: flaky-test-handler > My github account: qingzhouluo/seriousamlqz (Please add both, thanks). > > Description: > > This plugin is designed to handle flaky tests. Currently it aims for Git > and Maven. > > 1. We recently made some contribution to Maven Surefire to add the new > option -rereunFailingTestsCount. It was already accepted, and may be > released with 2.18 in the near future: > > > https://github.com/apache/maven-surefire/commit/fefaae7f0534a59f52c046a64c96987e8561dd48 > > It basically let users to choose to re-run failing tests up to N times, > and if it passes any of those N times, it will be marked as a "Flake" and > the build will be marked as successful. The generated test report will > contain each re-run information. > > So the first part of our plugin is to integrate with this new feature from > Maven Surefire. It will parse the report and display flaky tests, output > information of all the re-runs(stacktrace, output, etc) on test result page. > > https://drive.google.com/file/d/0Bzfyw2NYCLFBV2lnMVlrWVhueGc > https://drive.google.com/file/d/0Bzfyw2NYCLFBZXFxSThlMWlaeVU > > 2. For each failing build we provide a "Deflake" action. It is different > from rebuilding the project, as it will: > > 1) Checkout the exact revision of that failing build (GIT) > 2) Only run all those failed tests. > > So this will give developer a good idea about how many tests are flaky for > this build. > > https://drive.google.com/file/d/0Bzfyw2NYCLFBR3RDM3psQjhIRUk/ > https://drive.google.com/file/d/0Bzfyw2NYCLFBeTFjQk5sQW5XR3M/ > > 3. Aggregate information from both 1 and 2, and display them to the users. > > For each git revision and each test, we count how many times it passed, > how many times it failed. So if a test passed and failed for the both > revision, we count it as a flake; if it always failed, then it's a fail, > otherwise it's a pass. > > https://drive.google.com/file/d/0Bzfyw2NYCLFBRlI2SHpubTltd28/ > https://drive.google.com/file/d/0Bzfyw2NYCLFBVWJNWFAweEl1Y28/ > > So this help users to see what are my most flaky tests (they can compute > some kind of flakiness), so they can fix them first. It basically serves as > the test health page. > > We have a repo on github, however it was empty right now. > https://github.com/google/jenkins-flaky-test-report > > Will push our code to it very soon. So here I want to get a repo from > Jenkins so we can push our code to Jenkins too. > > Thank you very much! > > Regards, > > Qingzhou > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
