This sounds *very* interesting. I am currently working on doing at least some nightly work on one of our virtual servers. First shot will be at the doc, James is working with me. So far, we wait for the setup ;) I think it's best to focus a bit, so do the doc first and then look at the testing.
It would be even greater if you could join that effort once we are ready. Rainer On Wed, Feb 26, 2014 at 11:07 AM, Radu Gheorghe <[email protected]>wrote: > 2014-02-26 12:01 GMT+02:00 Radu Gheorghe <[email protected]>: > > Hello and excuse my late reply, > > > > I'm willing to help with polishing the ES tests I've already written, > > and I guess having new ones plugged in would be easy once we have a > > skeleton. > > > > As for the ES setup itself, I got an idea following David's comment > > about VMs. Because I'm working on a log analytics product called > > Logsene: > > Sorry, weird key combination sent the Email outside of my will. Back to > Logsene: > http://sematext.com/logsene/index.html > > The trick about it is that it exposes the ES API, so one can use it > for tests, proof of concepts, etc. We can have a free account with a > free-plan application there that we can use for testing. As long as > the build machine has Internet access, it doesn't have to have ES > installed, or install it in the first place. But the tests need to > have the Logsene application's token stored somewhere. Ideally, the > token should be kept private, otherwise anyone could mess up with our > tests. > > Best regards, > Radu > > > > > 2014-02-22 21:32 GMT+02:00 David Lang <[email protected]>: > >> I think we need two different things. > >> > >> 1. we need a stand-alone test capability so that people hacking the > code can > >> make sure that they haven't broken anything. > >> > >> This is what Radu mentioned. > >> > >> 2. we need an integration test to make sure that we work with the actual > >> systems (and possibly several versions) > >> > >> This is what Rainer mentioned. > >> > >> > >> > >> when people are hacking on the code, they may do things that break > >> processing or variable definitions, and we want a 'known data' test > that all > >> we do is a unit test to see if what we send to the remote socket matches > >> what we expect to send. This doesn't need to talk to a server that > >> understands the protocol, it just needs a "If we get string 1 reply with > >> string 2, if we get string 3 reply with string 4" dumb TCP responder. > >> > >> Ideally, we would require this sort of test when a module is submitted. > >> > >> > >> > >> Then as a separate test, we need to talk to Google/Amazon/other VM > provider > >> to see about having them donate VM time so that we can setup 'real' > >> destination systems, and then have a test harness that can clone a > 'known > >> good' target, run tests against the clone, then destroy the clone. > >> > >> So, there are a few things we would need to do: > >> > >> 1. build/find a test harness to run such a setup. > >> > >> 2. define what targets we need > >> > >> A. what versions of each target need to be tested. > >> > >> How many different versions of PostgreSQL are to be supported? > >> > >> 3. find 'owners' for each target type. > >> > >> This needs to be someone who understands the target to configure the > >> system, and to respond to issues when we pass the unit test against the > dumb > >> responder mentioned above, but fail against the real system. This will > >> probably involve tinkering with the unit test strings. > >> > >> > >> A. What should we do for target types that require paid licenses? > >> Ideally the company that owns that target type should be involved to > provide > >> us with the license (and some troubleshooting), but in some cases they > >> aren't going to care. If the vendor isn't going to provide the > licesnses, > >> who is going to pay for this? > >> > >> 4. Get funding/donations to cover VM time. > >> > >> David Lang > >> > >> > >> On Fri, 21 Feb 2014, Rainer Gerhards wrote: > >> > >>> Date: Fri, 21 Feb 2014 18:52:29 +0100 > >>> From: Rainer Gerhards <[email protected]> > >>> Reply-To: rsyslog-users <[email protected]> > >>> To: rsyslog-users <[email protected]> > >>> Subject: Re: [rsyslog] How should the testbench deal with DBs? > >>> > >>> > >>> Let me hijack this thread at least a little... > >>> > >>> First: the issue of multiple DB engines and such as real pain in the > b... > >>> for me. I'd like to have as many automated tests as possible, but all > >>> those > >>> plugins that need "elaborate" setup usually fall short as it is pretty > >>> hard > >>> to do (for me and on every dev machine). Remember that for some > plugins I > >>> don't even have the slightest idea of how the backend works. And for > sure > >>> not even remotely how I could conduct an automated test, which requires > >>> > >>> a) getting to a known initial state > >>> b) shuffeling data at a test run > >>> c) extracting the results and check if they are ok > >>> d) probably re-setting things to look untouched again (ok, somewhat > >>> redundant with a). > >>> > >>> Keeping the ES example, I know barely how I can do a, c, and d > manually so > >>> that I can conduct manual tests (and after some weeks this even > requires > >>> review of notes, as I really don't care/use ES in day-to-day > operations). > >>> Doing this in an automated way is way above my head. Now think about > the > >>> many other plugins, and those that hopefully will be written. I am not > >>> that > >>> super-guru, and I think for many folks it's hard to be really good at > all > >>> of these tools (congrats if you are!). > >>> > >>> Disabling a test if the feature is not enabled might help an > end-"user", > >>> but it doesn't solve the core problem that I don't detect regressions > >>> right > >>> in time. > >>> > >>> So... now to the hijacking. I have a dream. As a community, I think we > >>> would be able to create a machine where we could run nightly builds, > >>> nightly *testbench* runs (on a much more elaborate testbench including > all > >>> the backend system), night doc generation and upload ... and so on. > >>> > >>> This would need to be a group effort as we would need to drag in all > the > >>> experts that could help craft these tests. Also, we would need some > folks > >>> who would take care of handling the server itself, I mean the usual > admin > >>> things like keeping it secure, creating user accounts for other > >>> collaborators and so on. I know we have many site reliability engineers > >>> (and similar positions) on our list, so they could chime in here. Such > a > >>> machine would probably also enable me to find bugs quicker. Right now, > for > >>> things like ES that I don't work on regularly, the main obstacle to > start > >>> a > >>> bughunt is to setup the env, remember how get around with it and so > it. If > >>> there is a ready machine with clear procedure, I could hunt bugs much > more > >>> efficient. But quite frankly this is a too large task for us small > Adiscon > >>> to handle. We simply have no funding for all that talent required. > >>> Regarding the platform, I'd assume that a virtual server somewhere in > the > >>> cloud could be used (hopefully a sponsored one). And as I am right now > in > >>> "whish mode", it would help immensely to have access to maintained > build > >>> environments for Solaris, SmartOS, AIX, *BSD and so on, together with > some > >>> helping hands I could ask if I don't find may way around easily enough. > >>> > >>> OK, that was the dream. Does anybody think we could realize at least a > bit > >>> of it in practice? ;) > >>> > >>> Thank, > >>> Rainer > >>> > >>> > >>> On Fri, Feb 21, 2014 at 4:54 PM, Radu Gheorghe > >>> <[email protected]>wrote: > >>> > >>>> Hello increasingly-active-rsyslog-list! > >>>> > >>>> I think we need to decide how to move forward with the testbench, > >>>> whenever people find the time to work on it. The problem came from > >>>> comments of this issue: > >>>> https://github.com/rsyslog/rsyslog/issues/10 > >>>> > >>>> (check out the comments after the issue was closed) > >>>> > >>>> Basically, the general decision point is to see how we should handle > >>>> rsyslog's integrations with various data stores. Whether that's > >>>> Elasticsearch, MongoDB, MySQL, etc. > >>>> > >>>> So far, dealing with files was done by... wait for it... writing to > >>>> actual files. This gets more complicated with DBs because you need to > >>>> have the DB there. Installing it might take time and tons of > >>>> dependencies. > >>>> > >>>> There are a couple of solutions mentioned in the GitHub issue, but I'm > >>>> throwing a new one after a bit of thinking. Let's assume that the > >>>> likes of Elasticsearch are already there, and make a list of > >>>> "requirements" so that one can see what is assumed. > >>>> > >>>> The point is, if you don't have ES installed, you probably don't care > >>>> about ES tests. So all tests about ES could, for instance, try and do > >>>> a: > >>>> > >>>> curl localhost:9200 > >>>> > >>>> And if there's an error, or the version is too old, skip the test and > >>>> print a warning. > >>>> > >>>> What do you think? Other suggestions? > >>>> > >>>> Best regards, > >>>> Radu > >>>> > >>>> P.S. The point of starting this thread is, as you might see from > >>>> GitHub, that I've worked on a few omelasticsearch tests, and I'd like > >>>> to see what we need in order to get them into the main rsyslog > >>>> testbench. Blindly assuming ES is there is a bit too much, as Rainer > >>>> pointed out. > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com/professional-services/ > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> DON'T LIKE THAT. > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > >>> LIKE THAT. > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of > >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > >> LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
