On Friday, 5 June 2015 at 10:00:17 UTC, Rikki Cattermole wrote:
On 5/06/2015 8:28 p.m., Jonathan M Davis wrote:
On Friday, 5 June 2015 at 06:42:06 UTC, Rikki Cattermole wrote:
And shouldn't the CI be doing regression testing already?
The autotester runs the unit tests that are in druntime,
Phobos, and
dmd. It catches a lot of stuff and generally prevents us from
merging
bad code. But it can't possibly catch everything. It catches
the stuff
that we thought to specifically test for and bugs that were
fixed (since
unit tests are usually added for bug fixes). But it's not that
infrequent for something _new_ to break that's never broken
before and
is subtle enough that it doesn't get past all of the tests -
especially
when you're dealing with changes to the compiler.
To catch all of that stuff before it goes out the door in a
release, we
need to test a _much_ larger code base than just the standard
stuff -
which is part of why we have betas. We want folks to try out
their
projects with the betas so that we can catch the stuff that we
missed
before it gets released in an official release. Simply
grabbing an
arbitrary commit and declaring it a release just because it's
at about
the time that we want to do a release would be a disaster. Too
much gets
through as it is simply because not enough folks test the
betas and
report what they find. _All_ of that would get through if we
just picked
a random commit and declared it to be a release.
_Maybe_ someday our test suites will catch such a large
portion of the
regressions that we won't actually end up with any regressions
getting
out, but I doubt it. Even large, heavily used projects like
gcc or KDE
end up with regressions getting out, much as they try to avoid
it. But
like them, we need to do our best to have releases which have
been
tested well enough via betas and whatnot rather than just
releasing
stuff simply because it's a certain date.
- Jonathan M Davis
I wonder if we could integrate a bit more with travis and have
a big list of projects compiling + testing against HEAD.
Would be an excellent indicator.
Maybe even have this as part of dub repo?
I remember there being talk about testing HEAD against vibe.d, I
can't remember what happened(if anything.)
