Hello and excuse my late reply, I'm willing to help with polishing the ES tests I've already written, and I guess having new ones plugged in would be easy once we have a skeleton.
As for the ES setup itself, I got an idea following David's comment about VMs. Because I'm working on a log analytics product called Logsene: 2014-02-22 21:32 GMT+02:00 David Lang <[email protected]>: > I think we need two different things. > > 1. we need a stand-alone test capability so that people hacking the code can > make sure that they haven't broken anything. > > This is what Radu mentioned. > > 2. we need an integration test to make sure that we work with the actual > systems (and possibly several versions) > > This is what Rainer mentioned. > > > > when people are hacking on the code, they may do things that break > processing or variable definitions, and we want a 'known data' test that all > we do is a unit test to see if what we send to the remote socket matches > what we expect to send. This doesn't need to talk to a server that > understands the protocol, it just needs a "If we get string 1 reply with > string 2, if we get string 3 reply with string 4" dumb TCP responder. > > Ideally, we would require this sort of test when a module is submitted. > > > > Then as a separate test, we need to talk to Google/Amazon/other VM provider > to see about having them donate VM time so that we can setup 'real' > destination systems, and then have a test harness that can clone a 'known > good' target, run tests against the clone, then destroy the clone. > > So, there are a few things we would need to do: > > 1. build/find a test harness to run such a setup. > > 2. define what targets we need > > A. what versions of each target need to be tested. > > How many different versions of PostgreSQL are to be supported? > > 3. find 'owners' for each target type. > > This needs to be someone who understands the target to configure the > system, and to respond to issues when we pass the unit test against the dumb > responder mentioned above, but fail against the real system. This will > probably involve tinkering with the unit test strings. > > > A. What should we do for target types that require paid licenses? > Ideally the company that owns that target type should be involved to provide > us with the license (and some troubleshooting), but in some cases they > aren't going to care. If the vendor isn't going to provide the licesnses, > who is going to pay for this? > > 4. Get funding/donations to cover VM time. > > David Lang > > > On Fri, 21 Feb 2014, Rainer Gerhards wrote: > >> Date: Fri, 21 Feb 2014 18:52:29 +0100 >> From: Rainer Gerhards <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] How should the testbench deal with DBs? >> >> >> Let me hijack this thread at least a little... >> >> First: the issue of multiple DB engines and such as real pain in the b... >> for me. I'd like to have as many automated tests as possible, but all >> those >> plugins that need "elaborate" setup usually fall short as it is pretty >> hard >> to do (for me and on every dev machine). Remember that for some plugins I >> don't even have the slightest idea of how the backend works. And for sure >> not even remotely how I could conduct an automated test, which requires >> >> a) getting to a known initial state >> b) shuffeling data at a test run >> c) extracting the results and check if they are ok >> d) probably re-setting things to look untouched again (ok, somewhat >> redundant with a). >> >> Keeping the ES example, I know barely how I can do a, c, and d manually so >> that I can conduct manual tests (and after some weeks this even requires >> review of notes, as I really don't care/use ES in day-to-day operations). >> Doing this in an automated way is way above my head. Now think about the >> many other plugins, and those that hopefully will be written. I am not >> that >> super-guru, and I think for many folks it's hard to be really good at all >> of these tools (congrats if you are!). >> >> Disabling a test if the feature is not enabled might help an end-"user", >> but it doesn't solve the core problem that I don't detect regressions >> right >> in time. >> >> So... now to the hijacking. I have a dream. As a community, I think we >> would be able to create a machine where we could run nightly builds, >> nightly *testbench* runs (on a much more elaborate testbench including all >> the backend system), night doc generation and upload ... and so on. >> >> This would need to be a group effort as we would need to drag in all the >> experts that could help craft these tests. Also, we would need some folks >> who would take care of handling the server itself, I mean the usual admin >> things like keeping it secure, creating user accounts for other >> collaborators and so on. I know we have many site reliability engineers >> (and similar positions) on our list, so they could chime in here. Such a >> machine would probably also enable me to find bugs quicker. Right now, for >> things like ES that I don't work on regularly, the main obstacle to start >> a >> bughunt is to setup the env, remember how get around with it and so it. If >> there is a ready machine with clear procedure, I could hunt bugs much more >> efficient. But quite frankly this is a too large task for us small Adiscon >> to handle. We simply have no funding for all that talent required. >> Regarding the platform, I'd assume that a virtual server somewhere in the >> cloud could be used (hopefully a sponsored one). And as I am right now in >> "whish mode", it would help immensely to have access to maintained build >> environments for Solaris, SmartOS, AIX, *BSD and so on, together with some >> helping hands I could ask if I don't find may way around easily enough. >> >> OK, that was the dream. Does anybody think we could realize at least a bit >> of it in practice? ;) >> >> Thank, >> Rainer >> >> >> On Fri, Feb 21, 2014 at 4:54 PM, Radu Gheorghe >> <[email protected]>wrote: >> >>> Hello increasingly-active-rsyslog-list! >>> >>> I think we need to decide how to move forward with the testbench, >>> whenever people find the time to work on it. The problem came from >>> comments of this issue: >>> https://github.com/rsyslog/rsyslog/issues/10 >>> >>> (check out the comments after the issue was closed) >>> >>> Basically, the general decision point is to see how we should handle >>> rsyslog's integrations with various data stores. Whether that's >>> Elasticsearch, MongoDB, MySQL, etc. >>> >>> So far, dealing with files was done by... wait for it... writing to >>> actual files. This gets more complicated with DBs because you need to >>> have the DB there. Installing it might take time and tons of >>> dependencies. >>> >>> There are a couple of solutions mentioned in the GitHub issue, but I'm >>> throwing a new one after a bit of thinking. Let's assume that the >>> likes of Elasticsearch are already there, and make a list of >>> "requirements" so that one can see what is assumed. >>> >>> The point is, if you don't have ES installed, you probably don't care >>> about ES tests. So all tests about ES could, for instance, try and do >>> a: >>> >>> curl localhost:9200 >>> >>> And if there's an error, or the version is too old, skip the test and >>> print a warning. >>> >>> What do you think? Other suggestions? >>> >>> Best regards, >>> Radu >>> >>> P.S. The point of starting this thread is, as you might see from >>> GitHub, that I've worked on a few omelasticsearch tests, and I'd like >>> to see what we need in order to get them into the main rsyslog >>> testbench. Blindly assuming ES is there is a bit too much, as Rainer >>> pointed out. >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
