On 5/06/2015 8:28 p.m., Jonathan M Davis wrote:
On Friday, 5 June 2015 at 06:42:06 UTC, Rikki Cattermole wrote:
And shouldn't the CI be doing regression testing already?
The autotester runs the unit tests that are in druntime, Phobos, and
dmd. It catches a lot of stuff and generally prevents us from merging
bad code. But it can't possibly catch everything. It catches the stuff
that we thought to specifically test for and bugs that were fixed (since
unit tests are usually added for bug fixes). But it's not that
infrequent for something _new_ to break that's never broken before and
is subtle enough that it doesn't get past all of the tests - especially
when you're dealing with changes to the compiler.
To catch all of that stuff before it goes out the door in a release, we
need to test a _much_ larger code base than just the standard stuff -
which is part of why we have betas. We want folks to try out their
projects with the betas so that we can catch the stuff that we missed
before it gets released in an official release. Simply grabbing an
arbitrary commit and declaring it a release just because it's at about
the time that we want to do a release would be a disaster. Too much gets
through as it is simply because not enough folks test the betas and
report what they find. _All_ of that would get through if we just picked
a random commit and declared it to be a release.
_Maybe_ someday our test suites will catch such a large portion of the
regressions that we won't actually end up with any regressions getting
out, but I doubt it. Even large, heavily used projects like gcc or KDE
end up with regressions getting out, much as they try to avoid it. But
like them, we need to do our best to have releases which have been
tested well enough via betas and whatnot rather than just releasing
stuff simply because it's a certain date.
- Jonathan M Davis
I wonder if we could integrate a bit more with travis and have a big
list of projects compiling + testing against HEAD.
Would be an excellent indicator.
Maybe even have this as part of dub repo?
