On 12/14/2016 11:22 AM, Martin Buchholz wrote:
On Mon, Nov 28, 2016 at 4:50 PM, Joseph D. Darcy <joe.da...@oracle.com
<mailto:joe.da...@oracle.com>> wrote:
On 11/28/2016 3:30 PM, Martin Buchholz wrote:
As an aside, for JDK 10 I'd also like to see promoted builds
on a more frequent schedule than once a week.
People do "continuous testing and integration" these days. Set
up your integration pipeline so that it is always running. The
pipeline automatically promotes changesets to master when all
tests pass. Easy! Then every master changeset is equally stable
to a "promoted build".
One of the internal systems mentioned above is a CI system that
run regression tests after a change is pushed. The approximate
integration model is then to promote known-good states of sources
as vetted by the CI system.
If problems are fixed soon after they are identified, then a
post-push system gives good results with avoiding the need more
complexity on the front end to manage a series of in-flight patches.
A CI system available to every committer that would guarantee no
regressions would be awesome!
I fundamentally do not believe in post-submit testing. I want to be
protected from my own mistakes and those of others. Google has also
moved towards doing more presubmit testing.
For large low-level infrastructure projects like openjdk, I envision
different layers of testing. The first layer would run e.g. the jtreg
and jck tests, perhaps on a distributed test farm. A second layer
might run publicly available java test suites that would be too
expensive to run in a first layer. Promotion to a "golden master"
might be dependent on passing tests in the second layer. Of course,
that would be a serious investment in testing/quality.
The tiered testing efforts made earlier in JDK 9 [1] are necessary
preconditions for a JDK-wide pre-push/pre-submit system. In particular,
if you don't have a a set of meaningful tests that runs quickly enough
and passes reliably enough then a pre-push system can cause more harm
than good in introducing bottlenecks and causing changes to be
spuriously rejected, say by an existing intermittent failure unrelated
to the change.
If the testing preconditions are met, there is still the small matter of
the submission system managing the queues, perhaps some with kind of
scoreboarding algorithm (https://en.wikipedia.org/wiki/Scoreboarding) to
reduce latencies, etc.
In the meantime, I think post-push testing and prompt fixing is a fine
approximation to pre-push testing and something we can (continue to) do now.
Independent of pre-push or post-push, the most important aspect of this
is making sure everyone treats build and test failures as urgent
problems that need to be addressed quickly.
Cheers,
-Joe
[1] "Proposed new policies for JDK 9 regression tests: tiered testing,
intermittent failures, and randomness,"
http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html
“Proposed new policies for JDK 9 regression tests: tiered testing,
intermittent failures, and randomness,”
http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-April/002164.html
“Test policy follow-up, third testing tier,”
http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-June/002325.html
