On Friday, 5 June 2015 at 06:42:06 UTC, Rikki Cattermole wrote:
And shouldn't the CI be doing regression testing already?
The autotester runs the unit tests that are in druntime, Phobos, and dmd. It catches a lot of stuff and generally prevents us from merging bad code. But it can't possibly catch everything. It catches the stuff that we thought to specifically test for and bugs that were fixed (since unit tests are usually added for bug fixes). But it's not that infrequent for something _new_ to break that's never broken before and is subtle enough that it doesn't get past all of the tests - especially when you're dealing with changes to the compiler.
To catch all of that stuff before it goes out the door in a release, we need to test a _much_ larger code base than just the standard stuff - which is part of why we have betas. We want folks to try out their projects with the betas so that we can catch the stuff that we missed before it gets released in an official release. Simply grabbing an arbitrary commit and declaring it a release just because it's at about the time that we want to do a release would be a disaster. Too much gets through as it is simply because not enough folks test the betas and report what they find. _All_ of that would get through if we just picked a random commit and declared it to be a release.
_Maybe_ someday our test suites will catch such a large portion of the regressions that we won't actually end up with any regressions getting out, but I doubt it. Even large, heavily used projects like gcc or KDE end up with regressions getting out, much as they try to avoid it. But like them, we need to do our best to have releases which have been tested well enough via betas and whatnot rather than just releasing stuff simply because it's a certain date.
- Jonathan M Davis
