We already have make check being run daily. So it's not on every commit, but
given the time it takes to run and the pace of merging new features into the
master tree, daily should be just fine.
The problems that we run into are not easy for the system to find. As I
understand it, we have a few noticable pain points.
1. distro specific problems.
The CentOS boot problem for example.
The ongoing SELinux/etc issues that we run into are another
Systemd is going to be more common in the future (and it's a moving target)
2. difficulty in setting up a repeatable test environment.
Output to Oracle or ElasticSearch are two examples. The difficulty is setting
up the external data store in such a way that it can be populated,
tested, and then reset back to the known good state.
Other examples are connections to the various MQ systems.
3. (extending #2) Contributed modules tend to be especially bad here. They tend
to be written for specific cases and have heavy depenencies on external
components. Without assitance from someone who knows those external components
well, setting up a good test is _hard_
4. (extending #2) configuring modules can sometimes be hard.
The GnuTLS problems are a good example of this sort of thing. Even when tests
exist, that just means that those particular configuations work. People trying
to use different features of the module (or in this case, the underlying library
and certificates) run into new problems.
#1 needs more people running the tests, and just trying to use the system on a
variety of situations.
#4 needs documentation and more tests written.
Rainer does a pretty good job at testing things before they hit the master
branch. While I was working at Intuit building their logging systems out, I was
very comfortable running custom builds from git in production with just a short
amount of 'smoke test' time. I wish all projects had as good a track record.
We really don't have that many real bugs reported, and the vast majority of the
problems that we do run into tend to be race conditions and other things that
are fairly hard to trigger. That doesn't mean that we don't have occasional
problems. The segfault bug that a handful of people have reported recently is an
example of that, but the fact that it's so hard to duplicate means that
automated testing isn't likely to help a lot.
David Lang
On Fri, 31 Oct 2014, singh.janmejay
wrote:
Im unsure if I have understood you correctly, but you seem to be thinking
of CI as a way of writing tests(like a testing framework, eg. google-test,
make-check etc).
But actually CI is a process(not a tool/framework). I guess this is the
best source of information about it:
http://en.wikipedia.org/wiki/Continuous_integration
So having a CI process in simple terms means that the integration-branch
(which happens to be master in our case) and all the other long-lived
branches such as feature-branches that live for a few days/weeks, should be
monitored by an automated build process.
This build process would trigger 'make check' for every new commit (people
often use git-push based post-update hook to notify build-server). This
build would be reported on a CI dashboard, which tells people if the build
was green or red.
CI servers also go one step further and support output-artifact integration
so developers can see which tests failed, why they failed etc.
So no changes are required to the way rsyslog codebase is right now. We
have a directory full of tests, we have a command that returns non-zero
exit code for failing tests and this setup is sufficient for us to setup an
automated build process.
We should check if travis can run 'make check' for us.
On Thu, Oct 30, 2014 at 8:38 PM, Rainer Gerhards <[email protected]>
wrote:
2014-10-30 13:35 GMT+01:00 singh.janmejay <[email protected]>:
+1 for testsuite and CI.
I tried to get a quick glimpse at CI, but not only rsyslog doc is full of
acronyms and references that talk only for those in the field ;)
Can someone tell me in quick words how in CI a test to do this looks like:
- spin up a rsyslogd listener instance "LST"
- spin up a rsyslogd sender instance "SND"
- make "SND" send a couple of thousand messages to LST
- stop both instances *when work is finished*
- count if LST did receive all messages and did so in correct format
This is one of the basic tests in the current rsyslog testbench.
I would appreciate if a control file for CI could be provided, so that I
can judge the effort that's to be made if I were to convert to that system
(it sounds interesting). Is there some volunteer who would migrate the
current testbench and educate me ... in the way James has done this for the
rsyslog-doc project?
Thanks,
Rainer
+1 for time based releases.
It'll be valuable to have adhoc minor/micro releases. Release feature by
feature or few features/fixes at a time type of thing.
--
Regards,
Janmejay
PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.
On Oct 30, 2014 6:01 PM, "Thomas D." <[email protected]> wrote:
Hi,
+1 for an always working "master" branch.
Do your work in feature branches. Only merge when you think the changes
are ready. Don't merge when you think "I am ready, but this needs
testing".
Regarding the testing problem in general: I would stop adding new
features for a while. Spend more time in improving code quality.
Try to find/create a working test suite. There will always be the
problem that nobody will test your stuff. So you need a way to write
tests. Yes, creating a test suite will take some time. But in the end
it
will improve the software quality and boost your development.
Remember that you can use CI for free with GitHub. Every pull request
could be automatically tested for you...
+1 for a time-based release approach.
PS: Debian Jessie will freeze at 23:59 UTC on the 5th of November 2014.
Just a reminder if you want to see some of the recent changes in
Jessie...
-Thomas
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
