Let me get straight to the beef (nun pun intended): This sounds all really great. Is anyone willing to do the initial work of creating a couple of tests and integrate them into the rsyslog process?
Rainer 2014-10-31 7:35 GMT+01:00 singh.janmejay <[email protected]>: > Im trying to see these problems in the light of CI. > > On Fri, Oct 31, 2014 at 10:50 AM, David Lang <[email protected]> wrote: > > > We already have make check being run daily. So it's not on every commit, > > but given the time it takes to run and the pace of merging new features > > into the master tree, daily should be just fine. > > > > The problems that we run into are not easy for the system to find. As I > > understand it, we have a few noticable pain points. > > > > 1. distro specific problems. > > > > The CentOS boot problem for example. > > The ongoing SELinux/etc issues that we run into are another > > Systemd is going to be more common in the future (and it's a moving > > target) > > > > My team had to solve a similar problem with one of my ex-employers. Our CI > setup was composed of machines running various environments. We had > machines running CentOS, Windows, Solaris etc. All of these machines were > running CI-agent/worker and the build was configured to do exactly the same > thing on each worker. So we basically found issues before releasing > GA-versions provided we had all relevant versions of environments running. > This kind of infrastructure would require several machines/VMs, each one > running one supported environment. But this also requires some budget to > finance the setup. > > The only other option seems to be what -devel releases are supposed to > facilitate, but if we had limited success with that, it can't be counted as > a real option. > > > > > > 2. difficulty in setting up a repeatable test environment. > > > > Output to Oracle or ElasticSearch are two examples. The difficulty is > > setting up the external data store in such a way that it can be > populated, > > tested, and then reset back to the known good state. > > > > Other examples are connections to the various MQ systems. > > > > Common/recommended practise here is to make each tests stateless. Most > language/platform specific testing-frameworks provide a notion of setup and > teardown. Large end-to-end integration tests usually have elaborate > setup/teardown steps which involve truncating database tables, deleting > elasticsearch indices, draining/purging relevant external queues etc. > > But most value is derived from having 2 levels of tests. > > The make-check tests that we have are end-to-end integration tests. Such > tests generally face external-dependency based flakiness problems, until > setup/teardown are done very well (like they aggressively re-try failed > operations etc). > > A lower level of test would be unit-tests, which test individual functions, > or set of functions rather than testing end-to-end behaviour of rsyslog. > This is where tools such as google-test, cppunit etc come in. These tests > are usually quicker to write as they don't involve elaborate orchestration. > For instance, regex-property-extractor could have dedicated tests to see if > it handles groups well, it handles char-classes well etc. With such 5 - 10 > tests guarding different behaviours of regex-property-extractor, we should > have enough confidence that it works standalone. The only thing integration > test needs to check is, does it work well when used end-to-end, for which > one test with simple input and output exercising match and submatch > features would suffice. Trouble starts when one tries to test all > combination of features through integration tests, which is usually > prohibitively expensive (not just in terms of running-time, but > writing/maintaining overhead too). > > > > > > 3. (extending #2) Contributed modules tend to be especially bad here. > They > > tend to be written for specific cases and have heavy depenencies on > > external components. Without assitance from someone who knows those > > external components well, setting up a good test is _hard_ > > > > Yes, writing integration tests involving multiple systems is painful. The > solution is to bank a lot more on unit-tests for functional correctness and > only write a few end-to-end tests to validate that integration points work > well with each other. Such a test would fail when a new version of > elasticsearch changes semantics of a certain API, while all unit-tests > would keep passing. > > Also, large amount of functional-testing can be done without involving > external systems by using mock-endpoints. For instance, think of having an > endpoint which receives elasticsearch bulk api payload and lets you > validate it was well-formed and semantically correct. This test won't fail > if elasticsearch changes its API, but it'll fail if anything in > implementation misbehaves, allowing one to catch large set of bugs easily. > End-to-end integration tests are then only required for the last-mile. > > > > > > 4. (extending #2) configuring modules can sometimes be hard. > > > > The GnuTLS problems are a good example of this sort of thing. Even when > > tests exist, that just means that those particular configuations work. > > People trying to use different features of the module (or in this case, > the > > underlying library and certificates) run into new problems. > > > > > Again, unit-tests allow one to test one feature standalone very easily. So > generally I'd write tests for each feature working alone involving minimal > set of features, and then cover bad combinations as they surface(or as we > expect them to surface). > > > > > > #1 needs more people running the tests, and just trying to use the system > > on a variety of situations. > > > > #4 needs documentation and more tests written. > > > > Rainer does a pretty good job at testing things before they hit the > master > > branch. While I was working at Intuit building their logging systems > out, I > > was very comfortable running custom builds from git in production with > just > > a short amount of 'smoke test' time. I wish all projects had as good a > > track record. > > > > We really don't have that many real bugs reported, and the vast majority > > of the problems that we do run into tend to be race conditions and other > > things that are fairly hard to trigger. That doesn't mean that we don't > > have occasional problems. The segfault bug that a handful of people have > > reported recently is an example of that, but the fact that it's so hard > to > > duplicate means that automated testing isn't likely to help a lot. > > > > I have had some amount of success in guarding regressions with > race-conditions in the past. It generally involves one big ugly test per > race-condition which uses semaphores and wait/notify mechanisms to make > code run in a certain way which is known to reproduce the problem. However, > this won't help in discovering potential/new race-conditions. In TDD > (test-driven-development) style of working, people write tests for new > features that involve complex concurrency concerns before/while writing > actual code. The idea is to enumerate as many race-conditions as possible > and cover them with tests. This approach is not bullet-proof, but it works > in most cases. > > > > > > David Lang > > > > > > On Fri, 31 Oct 2014, singh.janmejay wrote: > > > > Im unsure if I have understood you correctly, but you seem to be > thinking > >> of CI as a way of writing tests(like a testing framework, eg. > google-test, > >> make-check etc). > >> > >> But actually CI is a process(not a tool/framework). I guess this is the > >> best source of information about it: > >> http://en.wikipedia.org/wiki/Continuous_integration > >> > >> So having a CI process in simple terms means that the integration-branch > >> (which happens to be master in our case) and all the other long-lived > >> branches such as feature-branches that live for a few days/weeks, should > >> be > >> monitored by an automated build process. > >> > >> This build process would trigger 'make check' for every new commit > (people > >> often use git-push based post-update hook to notify build-server). This > >> build would be reported on a CI dashboard, which tells people if the > build > >> was green or red. > >> > >> CI servers also go one step further and support output-artifact > >> integration > >> so developers can see which tests failed, why they failed etc. > >> > >> So no changes are required to the way rsyslog codebase is right now. We > >> have a directory full of tests, we have a command that returns non-zero > >> exit code for failing tests and this setup is sufficient for us to setup > >> an > >> automated build process. > >> > >> We should check if travis can run 'make check' for us. > >> > >> > >> On Thu, Oct 30, 2014 at 8:38 PM, Rainer Gerhards < > >> [email protected]> > >> wrote: > >> > >> 2014-10-30 13:35 GMT+01:00 singh.janmejay <[email protected]>: > >>> > >>> +1 for testsuite and CI. > >>>> > >>>> > >>> I tried to get a quick glimpse at CI, but not only rsyslog doc is full > of > >>> acronyms and references that talk only for those in the field ;) > >>> > >>> Can someone tell me in quick words how in CI a test to do this looks > >>> like: > >>> > >>> - spin up a rsyslogd listener instance "LST" > >>> - spin up a rsyslogd sender instance "SND" > >>> - make "SND" send a couple of thousand messages to LST > >>> - stop both instances *when work is finished* > >>> - count if LST did receive all messages and did so in correct format > >>> > >>> This is one of the basic tests in the current rsyslog testbench. > >>> > >>> I would appreciate if a control file for CI could be provided, so that > I > >>> can judge the effort that's to be made if I were to convert to that > >>> system > >>> (it sounds interesting). Is there some volunteer who would migrate the > >>> current testbench and educate me ... in the way James has done this for > >>> the > >>> rsyslog-doc project? > >>> > >>> Thanks, > >>> Rainer > >>> > >>> > >>> +1 for time based releases. > >>>> > >>>> It'll be valuable to have adhoc minor/micro releases. Release feature > by > >>>> feature or few features/fixes at a time type of thing. > >>>> > >>>> -- > >>>> Regards, > >>>> Janmejay > >>>> > >>>> PS: Please blame the typos in this mail on my phone's uncivilized soft > >>>> keyboard sporting it's not-so-smart-assist technology. > >>>> > >>>> On Oct 30, 2014 6:01 PM, "Thomas D." <[email protected]> wrote: > >>>> > >>>> Hi, > >>>>> > >>>>> +1 for an always working "master" branch. > >>>>> > >>>>> Do your work in feature branches. Only merge when you think the > changes > >>>>> are ready. Don't merge when you think "I am ready, but this needs > >>>>> > >>>> testing". > >>>> > >>>>> > >>>>> Regarding the testing problem in general: I would stop adding new > >>>>> features for a while. Spend more time in improving code quality. > >>>>> Try to find/create a working test suite. There will always be the > >>>>> problem that nobody will test your stuff. So you need a way to write > >>>>> tests. Yes, creating a test suite will take some time. But in the end > >>>>> > >>>> it > >>> > >>>> will improve the software quality and boost your development. > >>>>> Remember that you can use CI for free with GitHub. Every pull request > >>>>> could be automatically tested for you... > >>>>> > >>>>> > >>>>> +1 for a time-based release approach. > >>>>> > >>>>> > >>>>> PS: Debian Jessie will freeze at 23:59 UTC on the 5th of November > 2014. > >>>>> Just a reminder if you want to see some of the recent changes in > >>>>> > >>>> Jessie... > >>>> > >>>>> > >>>>> > >>>>> -Thomas > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com/professional-services/ > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>> > >>>> myriad > >>> > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>>> DON'T LIKE THAT. > >>>>> > >>>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com/professional-services/ > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> DON'T LIKE THAT. > >>>> > >>>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> > >>> > >> > >> > >> > >> _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > > > > -- > Regards, > Janmejay > http://codehunk.wordpress.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
