Dear EasyBuilders,
For a couple of weeks now, we have been evaluating Travis
(https://travis-ci.org/) as an alternative to the Jenkins server hosted
at the ICT department of Ghent University for testing the pull requests
and the master/develop branches of the EasyBuild repositories on GitHub.
(note: this relates to the automated tests concerning only the unit test
suites, not the actual build tests & test reports for easyconfig pull
requests which a triggered by humans using 'eb --from-pr <PR#>
--upload-test-report')
Looking into Travis as an alternative to Jenkins was motivated by:
* various issues with Jenkins, going from being unavailable from time to
time due to all sorts of issues (usually things breaking with an update
of either Jenkins or one of the plugins we're using) to being quite slow
because the Java powering Jenkins was hogging all the CPU for some reason
* limited resources (4 workers), while the Jenkins setup is being used
for testing a multitude of other projects as well
* hard to maintain test configurations, and even almost-duplicate test
configurations that have grown/diverged historically
* limited availability (in time) of the tests reports for failing tests
* somewhat silly protective measures like having to approve the testing
of pull requests for contributors who are not white-listed yet
I was seriously getting fed up with these problems/limitations,
especially because they were impacting incoming contributions more and more.
A couple of weeks ago, I spend some time on making myself familiar with
Travis and setting up test configurations,
by defining a .travis.yml file in the develop branch of each of the
GitHub repositories (cfr. [1-3]).
Although there were a couple of small hick-ups initially, Travis turned
out to work out really well.
Not only were the test configurations now included in the repository
itself (and thus also under version control),
test results for pull requests were usually delivered a lot faster
through Travis compared to Jenkins.
In addition, we were consistently running the tests in more different
configurations compared to the Jenkins setup, with very little effort...
Moreover, this enables others (you, for example) to trivially evaluate
changes before opening a pull request, which is particularly useful for
the EasyBuild framework.
It suffices to create an account on https://travis-ci.org, and enable
testing for your forks of the EasyBuild repositories via the Travis
profile page [4].
Through the settings tab, you can configure Travis to test both changes
pushed to branches in your fork, and/or pull requests to your fork [5].
The Travis UI takes some getting used to, and certainly isn't perfect,
but is a lot better than the somewhat outdated Jenkins UI (see for
example [6]).
The intention was to use both Jenkins and Travis side-by-side for a
while (more-or-less until after the next EasyBuild release, v2.8.0),
but today another issue arose with an update of Jenkins that made me
pull the plug (for the EasyBuild repositories), cfr. [7].
Just now, I've disabled the GitHub hooks & services related to our
Jenkins setup, meaning that it will no longer be triggered to test pull
requests (or merges).
We will solely rely on Travis from now on for testing incoming
contributions.
The switch to Travis should not affect contributors in a negative way;
so far, I only see improvements.
A significant difference is probably that Travis doesn't go and dump
comments for every test result (pass or fail) in pull requests.
which implies no e-mails notifying you of the test result (some people
will be quite happy with that part alone)...
Travis does send notifications when you push a new branch to your own
fork, however, and when the tests fail in a branch of your own fork (if
testing of branches is enabled).
For now, I consider that a good thing, but I'm sure there's a way to
re-enable (e-mail) push notifications if we really want them back.
I suspect the biggest impact is going to be on the people reviewing
incoming pull requests, since they may have to notify a contributor of
failing tests...
I'm happy to answer any questions you may have on this matter.
If you notice anything strange or wrong with the Travis testing process,
please inform us in one way or another (email, IRC, ...).
regards,
Kenneth
[1] https://github.com/hpcugent/easybuild-framework/pull/1733: initial
.travis.yml for easybuild-framework
[2] https://github.com/hpcugent/easybuild-easyblocks/pull/895,
https://github.com/hpcugent/easybuild-easyblocks/pull/897
[3] https://github.com/hpcugent/easybuild-easyconfigs/pull/2942,
https://github.com/hpcugent/easybuild-easyconfigs/pull/2944
[4] https://travis-ci.org/profile
[5] https://travis-ci.org/YOUR_GITHUB_LOGIN/easybuild-framework/settings
[6] https://travis-ci.org/hpcugent/easybuild-framework
[7] https://issues.jenkins-ci.org/browse/JENKINS-34762
