Meant to throw this out yesterday direct to Rainer since it's not a quick/easy solution but the first time you see it (though most likely already have) it's sort of an "wow" moment...
http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-ap ache-mesos-part-i http://www.ebaytechblog.com/2014/05/12/delivering-ebays-ci-solution-with-ap ache-mesos-part-ii http://www.youtube.com/watch?v=VZPbLUJnR68 We are just working on this internally ourselves as part of a larger cloud initiative including docker. It's a lot to come up to speed on from scratch, but since beside CI/CD I routinely hear "it's hard to setup environments" this could pay huge dividends later...since you would just leave the underlying bare metal/VMs alone and deploy/start/stop/delete containers as needed. Docker files can be tracked in source control, and builds fired on commits, so end-to-end pipeline is easy once the plumbing exists. Unfortunately I can't volunteer anything yet, because I've just moved past learning phase and putting together a project plan before starting a PoC...after that I might have something more useful than the above links to share. :-) -----Original Message----- From: Rainer Gerhards <[email protected]> Reply-To: rsyslog-users <[email protected]> Date: Friday, October 31, 2014 at 3:53 AM To: rsyslog-users <[email protected]> Subject: Re: [rsyslog] CI - was: Feedback Request: do we still need -devel versions? >Let me get straight to the beef (nun pun intended): This sounds all really >great. Is anyone willing to do the initial work of creating a couple of >tests and integrate them into the rsyslog process? > >Rainer > >2014-10-31 7:35 GMT+01:00 singh.janmejay <[email protected]>: > >> Im trying to see these problems in the light of CI. >> >> On Fri, Oct 31, 2014 at 10:50 AM, David Lang <[email protected]> wrote: >> >> > We already have make check being run daily. So it's not on every >>commit, >> > but given the time it takes to run and the pace of merging new >>features >> > into the master tree, daily should be just fine. >> > >> > The problems that we run into are not easy for the system to find. As >>I >> > understand it, we have a few noticable pain points. >> > >> > 1. distro specific problems. >> > >> > The CentOS boot problem for example. >> > The ongoing SELinux/etc issues that we run into are another >> > Systemd is going to be more common in the future (and it's a moving >> > target) >> > >> >> My team had to solve a similar problem with one of my ex-employers. Our >>CI >> setup was composed of machines running various environments. We had >> machines running CentOS, Windows, Solaris etc. All of these machines >>were >> running CI-agent/worker and the build was configured to do exactly the >>same >> thing on each worker. So we basically found issues before releasing >> GA-versions provided we had all relevant versions of environments >>running. >> This kind of infrastructure would require several machines/VMs, each one >> running one supported environment. But this also requires some budget to >> finance the setup. >> >> The only other option seems to be what -devel releases are supposed to >> facilitate, but if we had limited success with that, it can't be >>counted as >> a real option. >> >> >> > >> > 2. difficulty in setting up a repeatable test environment. >> > >> > Output to Oracle or ElasticSearch are two examples. The difficulty >>is >> > setting up the external data store in such a way that it can be >> populated, >> > tested, and then reset back to the known good state. >> > >> > Other examples are connections to the various MQ systems. >> > >> >> Common/recommended practise here is to make each tests stateless. Most >> language/platform specific testing-frameworks provide a notion of setup >>and >> teardown. Large end-to-end integration tests usually have elaborate >> setup/teardown steps which involve truncating database tables, deleting >> elasticsearch indices, draining/purging relevant external queues etc. >> >> But most value is derived from having 2 levels of tests. >> >> The make-check tests that we have are end-to-end integration tests. Such >> tests generally face external-dependency based flakiness problems, until >> setup/teardown are done very well (like they aggressively re-try failed >> operations etc). >> >> A lower level of test would be unit-tests, which test individual >>functions, >> or set of functions rather than testing end-to-end behaviour of rsyslog. >> This is where tools such as google-test, cppunit etc come in. These >>tests >> are usually quicker to write as they don't involve elaborate >>orchestration. >> For instance, regex-property-extractor could have dedicated tests to >>see if >> it handles groups well, it handles char-classes well etc. With such 5 - >>10 >> tests guarding different behaviours of regex-property-extractor, we >>should >> have enough confidence that it works standalone. The only thing >>integration >> test needs to check is, does it work well when used end-to-end, for >>which >> one test with simple input and output exercising match and submatch >> features would suffice. Trouble starts when one tries to test all >> combination of features through integration tests, which is usually >> prohibitively expensive (not just in terms of running-time, but >> writing/maintaining overhead too). >> >> >> > >> > 3. (extending #2) Contributed modules tend to be especially bad here. >> They >> > tend to be written for specific cases and have heavy depenencies on >> > external components. Without assitance from someone who knows those >> > external components well, setting up a good test is _hard_ >> > >> >> Yes, writing integration tests involving multiple systems is painful. >>The >> solution is to bank a lot more on unit-tests for functional correctness >>and >> only write a few end-to-end tests to validate that integration points >>work >> well with each other. Such a test would fail when a new version of >> elasticsearch changes semantics of a certain API, while all unit-tests >> would keep passing. >> >> Also, large amount of functional-testing can be done without involving >> external systems by using mock-endpoints. For instance, think of having >>an >> endpoint which receives elasticsearch bulk api payload and lets you >> validate it was well-formed and semantically correct. This test won't >>fail >> if elasticsearch changes its API, but it'll fail if anything in >> implementation misbehaves, allowing one to catch large set of bugs >>easily. >> End-to-end integration tests are then only required for the last-mile. >> >> >> > >> > 4. (extending #2) configuring modules can sometimes be hard. >> > >> > The GnuTLS problems are a good example of this sort of thing. Even >>when >> > tests exist, that just means that those particular configuations work. >> > People trying to use different features of the module (or in this >>case, >> the >> > underlying library and certificates) run into new problems. >> > >> > >> Again, unit-tests allow one to test one feature standalone very easily. >>So >> generally I'd write tests for each feature working alone involving >>minimal >> set of features, and then cover bad combinations as they surface(or as >>we >> expect them to surface). >> >> >> > >> > #1 needs more people running the tests, and just trying to use the >>system >> > on a variety of situations. >> > >> > #4 needs documentation and more tests written. >> > >> > Rainer does a pretty good job at testing things before they hit the >> master >> > branch. While I was working at Intuit building their logging systems >> out, I >> > was very comfortable running custom builds from git in production with >> just >> > a short amount of 'smoke test' time. I wish all projects had as good a >> > track record. >> > >> > We really don't have that many real bugs reported, and the vast >>majority >> > of the problems that we do run into tend to be race conditions and >>other >> > things that are fairly hard to trigger. That doesn't mean that we >>don't >> > have occasional problems. The segfault bug that a handful of people >>have >> > reported recently is an example of that, but the fact that it's so >>hard >> to >> > duplicate means that automated testing isn't likely to help a lot. >> > >> >> I have had some amount of success in guarding regressions with >> race-conditions in the past. It generally involves one big ugly test per >> race-condition which uses semaphores and wait/notify mechanisms to make >> code run in a certain way which is known to reproduce the problem. >>However, >> this won't help in discovering potential/new race-conditions. In TDD >> (test-driven-development) style of working, people write tests for new >> features that involve complex concurrency concerns before/while writing >> actual code. The idea is to enumerate as many race-conditions as >>possible >> and cover them with tests. This approach is not bullet-proof, but it >>works >> in most cases. >> >> >> > >> > David Lang >> > >> > >> > On Fri, 31 Oct 2014, singh.janmejay wrote: >> > >> > Im unsure if I have understood you correctly, but you seem to be >> thinking >> >> of CI as a way of writing tests(like a testing framework, eg. >> google-test, >> >> make-check etc). >> >> >> >> But actually CI is a process(not a tool/framework). I guess this is >>the >> >> best source of information about it: >> >> http://en.wikipedia.org/wiki/Continuous_integration >> >> >> >> So having a CI process in simple terms means that the >>integration-branch >> >> (which happens to be master in our case) and all the other long-lived >> >> branches such as feature-branches that live for a few days/weeks, >>should >> >> be >> >> monitored by an automated build process. >> >> >> >> This build process would trigger 'make check' for every new commit >> (people >> >> often use git-push based post-update hook to notify build-server). >>This >> >> build would be reported on a CI dashboard, which tells people if the >> build >> >> was green or red. >> >> >> >> CI servers also go one step further and support output-artifact >> >> integration >> >> so developers can see which tests failed, why they failed etc. >> >> >> >> So no changes are required to the way rsyslog codebase is right now. >>We >> >> have a directory full of tests, we have a command that returns >>non-zero >> >> exit code for failing tests and this setup is sufficient for us to >>setup >> >> an >> >> automated build process. >> >> >> >> We should check if travis can run 'make check' for us. >> >> >> >> >> >> On Thu, Oct 30, 2014 at 8:38 PM, Rainer Gerhards < >> >> [email protected]> >> >> wrote: >> >> >> >> 2014-10-30 13:35 GMT+01:00 singh.janmejay >><[email protected]>: >> >>> >> >>> +1 for testsuite and CI. >> >>>> >> >>>> >> >>> I tried to get a quick glimpse at CI, but not only rsyslog doc is >>full >> of >> >>> acronyms and references that talk only for those in the field ;) >> >>> >> >>> Can someone tell me in quick words how in CI a test to do this looks >> >>> like: >> >>> >> >>> - spin up a rsyslogd listener instance "LST" >> >>> - spin up a rsyslogd sender instance "SND" >> >>> - make "SND" send a couple of thousand messages to LST >> >>> - stop both instances *when work is finished* >> >>> - count if LST did receive all messages and did so in correct format >> >>> >> >>> This is one of the basic tests in the current rsyslog testbench. >> >>> >> >>> I would appreciate if a control file for CI could be provided, so >>that >> I >> >>> can judge the effort that's to be made if I were to convert to that >> >>> system >> >>> (it sounds interesting). Is there some volunteer who would migrate >>the >> >>> current testbench and educate me ... in the way James has done this >>for >> >>> the >> >>> rsyslog-doc project? >> >>> >> >>> Thanks, >> >>> Rainer >> >>> >> >>> >> >>> +1 for time based releases. >> >>>> >> >>>> It'll be valuable to have adhoc minor/micro releases. Release >>feature >> by >> >>>> feature or few features/fixes at a time type of thing. >> >>>> >> >>>> -- >> >>>> Regards, >> >>>> Janmejay >> >>>> >> >>>> PS: Please blame the typos in this mail on my phone's uncivilized >>soft >> >>>> keyboard sporting it's not-so-smart-assist technology. >> >>>> >> >>>> On Oct 30, 2014 6:01 PM, "Thomas D." <[email protected]> wrote: >> >>>> >> >>>> Hi, >> >>>>> >> >>>>> +1 for an always working "master" branch. >> >>>>> >> >>>>> Do your work in feature branches. Only merge when you think the >> changes >> >>>>> are ready. Don't merge when you think "I am ready, but this needs >> >>>>> >> >>>> testing". >> >>>> >> >>>>> >> >>>>> Regarding the testing problem in general: I would stop adding new >> >>>>> features for a while. Spend more time in improving code quality. >> >>>>> Try to find/create a working test suite. There will always be the >> >>>>> problem that nobody will test your stuff. So you need a way to >>write >> >>>>> tests. Yes, creating a test suite will take some time. But in the >>end >> >>>>> >> >>>> it >> >>> >> >>>> will improve the software quality and boost your development. >> >>>>> Remember that you can use CI for free with GitHub. Every pull >>request >> >>>>> could be automatically tested for you... >> >>>>> >> >>>>> >> >>>>> +1 for a time-based release approach. >> >>>>> >> >>>>> >> >>>>> PS: Debian Jessie will freeze at 23:59 UTC on the 5th of November >> 2014. >> >>>>> Just a reminder if you want to see some of the recent changes in >> >>>>> >> >>>> Jessie... >> >>>> >> >>>>> >> >>>>> >> >>>>> -Thomas >> >>>>> _______________________________________________ >> >>>>> rsyslog mailing list >> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>>> http://www.rsyslog.com/professional-services/ >> >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>>>> >> >>>> myriad >> >>> >> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>you >> >>>>> DON'T LIKE THAT. >> >>>>> >> >>>>> _______________________________________________ >> >>>> rsyslog mailing list >> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>> http://www.rsyslog.com/professional-services/ >> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>you >> >>>> DON'T LIKE THAT. >> >>>> >> >>>> _______________________________________________ >> >>> rsyslog mailing list >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>> http://www.rsyslog.com/professional-services/ >> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>you >> >>> DON'T LIKE THAT. >> >>> >> >>> >> >> >> >> >> >> >> >> _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T LIKE THAT. >> > >> >> >> >> -- >> Regards, >> Janmejay >> http://codehunk.wordpress.com >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >_______________________________________________ >rsyslog mailing list >http://lists.adiscon.net/mailman/listinfo/rsyslog >http://www.rsyslog.com/professional-services/ >What's up with rsyslog? Follow https://twitter.com/rgerhards >NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
