On Wed, Dec 14, 2016 at 5:05 PM, joe darcy <joe.da...@oracle.com> wrote:
> > The tiered testing efforts made earlier in JDK 9 [1] are necessary > preconditions for a JDK-wide pre-push/pre-submit system. In particular, if > you don't have a a set of meaningful tests that runs quickly enough and > passes reliably enough then a pre-push system can cause more harm than good > in introducing bottlenecks and causing changes to be spuriously rejected, > say by an existing intermittent failure unrelated to the change. > It's a hard problem, but I think with sufficient effort presubmit can work well. Testing is embarrassingly parallel in principle so with enough hardware (or one of those "clouds" everyone is selling these days) one should be able to run arbitrarily many tests. No one seems to deal well with flaky tests, but flakiness can be measured, and a failing test can be rerun arbitrarily, so deflaking should be automatable. At Google we are starting to experiment with running jtreg tests with massive parallelism. Automation, presubmit testing and never-ever-broken master remain my goals for any software project.
