…
IMO, the unit testing patches shouldn't have been merged into the
release branch
I agree. This patch was in retrospective clearly not ready for a release
branch. A lot of people spend time to hot fix a broken build.
My root cause analysis boils down to:
Developers cannot detect multi-platform (build/run) issues with an
appropriate effort (or at all).
This is probably correct: the codebase is complex enough to cause
breakage on many types of changes, no matter how carefully the code is
reviewed. This is often because of the sheer number of options and their
invisible interplay. A recent example is the recent persist-tun / WFP
filtering issue, which slipped through all testing and review.
Regression testing using unit tests could possibly help us prevent this
type of breakages.
We already have discussed solutions to this:
1) Enable developers to run the “authoritative” buildslave tests before
submitting patches
The buildmaster can trigger manual builds from a different branch (e.g.
"staging/somebody"). Access to the buildmaster can be granted
selectively to trusted core developers. Non-trivial patches from other
people should probably be tested by one of these core developers before
pushing them to "master". Or we can just accept the fact that "master"
might be broken occasionally and fix the problems promptly.
2) Provide developers tooling to quickly (preferred: locally) run
iterations while developing
The Vagrant approach is nice because it will eventually allow developers
to run a fairly extensive smoketests on several various operating
systems with minimal effort:
<https://github.com/OpenVPN/openvpn/pull/45>
Right now the OS coverage is fairly minimal, but more variants can be
added easily, and we're not limited to Linux/Ubuntu like with Travis.
#1 will need some processes & tooling by the current maintainers.
#2 will be taken care of with the Vagrant based integration tests.
Opinions?
In addition to #1 and #2 we have Travis-CI smoke-testing in the pipeline:
<https://github.com/OpenVPN/openvpn/pull/52>
Travis-CI gives us some confidence on the quality of GitHub PRs, but
it's test matrix is much narrower than that of Buildbot, so it's
definitely not a panacea.
The only real way to test the more tricky corner-cases is to make it as
easy as possible to use Git "master" -based builds. For Windows this is
already covered by the Windows "buildslave" and the snapshots it
generates. I can setup apt repositories with Debian/Ubuntu builds based
on Git "master", but right now I don't have enough time to commit to that.
--
Samuli Seppänen
Community Manager
OpenVPN Technologies, Inc
irc freenode net: mattock
