On 11/28/2016 3:30 PM, Martin Buchholz wrote:
On Mon, Nov 28, 2016 at 2:07 PM, Joseph D. Darcy <joe.da...@oracle.com
<mailto:joe.da...@oracle.com>> wrote:
For the combined dev/master forest, the most recent integration
tag will have the same stability guarantees we have today so "pull
the most recent jdk-10+XYZ tag" to get a stable snapshot.
But ... I want better than the stability guarantees we have today!
Over the course of JDK 9 we've made marked improvements to the
regression test stability, at least on metrics we track internally on
internal systems, so unfortunately am I not at liberty to share them.
Briefly, the tier 1 regression tests in promoted builds are very stable,
the tier 2 tests somewhat less stable, and so on.
I want to obsolete those messages to quality-discuss with "new
failures" because it should not even be possible for master to get
into a state where regression tests have regressed (except for flakes
or external dependencies).
I don't know if those particular failures are actually in the promoted
build or just an artifact of how that particular batch of tests was run.
IIRC, we did not see a corresponding set of failures before the integration.
As an aside, for JDK 10 I'd also like to see promoted builds on a
more frequent schedule than once a week.
People do "continuous testing and integration" these days. Set up
your integration pipeline so that it is always running. The pipeline
automatically promotes changesets to master when all tests pass.
Easy! Then every master changeset is equally stable to a "promoted
build".
One of the internal systems mentioned above is a CI system that run
regression tests after a change is pushed. The approximate integration
model is then to promote known-good states of sources as vetted by the
CI system.
If problems are fixed soon after they are identified, then a post-push
system gives good results with avoiding the need more complexity on the
front end to manage a series of in-flight patches.
Cheers,
-Joe
