Really only opening a PR for this to gather some thoughts on this approach to dealing with flaky tests. It's easy to configure and makes use of out-of-the-box functionality of TestNG, but _does_ introduce the overhead of running this test three times on each run.
An approach an I also considered (similar to what's discussed in [this post](http://stackoverflow.com/questions/7803691/how-to-optimize-testng-and-seleniums-tests)) was to create a custom retry analyzer that would only run the test three times _if the first invocation failed_. Given the relatively small number of consistently flaky tests we have, though, my thought was first to try the out-of-the-box approach, and to go for the custom code if we feel the total overhead imposed by running the flaky tests multiple times on each run gets too big. You can merge this Pull Request by running: git pull https://github.com/jclouds/jclouds add-test-retry Or you can view, comment on it, or merge it online at: https://github.com/jclouds/jclouds/pull/577 -- Commit Summary -- * Expecting 2 of 3 or the first flaky test to succeed -- File Changes -- M core/src/test/java/org/jclouds/rest/functions/PresentWhenApiVersionLexicographicallyAtOrAfterSinceApiVersionTest.java (1) -- Patch Links -- https://github.com/jclouds/jclouds/pull/577.patch https://github.com/jclouds/jclouds/pull/577.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577
