Join what effort? Of hooking up you guys with Logsene? That would be my pleasure :D
On Wed, Feb 26, 2014 at 7:02 PM, Rainer Gerhards <[email protected]> wrote: > This sounds *very* interesting. I am currently working on doing at least > some nightly work on one of our virtual servers. First shot will be at the > doc, James is working with me. So far, we wait for the setup ;) I think > it's best to focus a bit, so do the doc first and then look at the testing. > > It would be even greater if you could join that effort once we are ready. > > Rainer > > > On Wed, Feb 26, 2014 at 11:07 AM, Radu Gheorghe > <[email protected]>wrote: > >> 2014-02-26 12:01 GMT+02:00 Radu Gheorghe <[email protected]>: >> > Hello and excuse my late reply, >> > >> > I'm willing to help with polishing the ES tests I've already written, >> > and I guess having new ones plugged in would be easy once we have a >> > skeleton. >> > >> > As for the ES setup itself, I got an idea following David's comment >> > about VMs. Because I'm working on a log analytics product called >> > Logsene: >> >> Sorry, weird key combination sent the Email outside of my will. Back to >> Logsene: >> http://sematext.com/logsene/index.html >> >> The trick about it is that it exposes the ES API, so one can use it >> for tests, proof of concepts, etc. We can have a free account with a >> free-plan application there that we can use for testing. As long as >> the build machine has Internet access, it doesn't have to have ES >> installed, or install it in the first place. But the tests need to >> have the Logsene application's token stored somewhere. Ideally, the >> token should be kept private, otherwise anyone could mess up with our >> tests. >> >> Best regards, >> Radu >> >> > >> > 2014-02-22 21:32 GMT+02:00 David Lang <[email protected]>: >> >> I think we need two different things. >> >> >> >> 1. we need a stand-alone test capability so that people hacking the >> code can >> >> make sure that they haven't broken anything. >> >> >> >> This is what Radu mentioned. >> >> >> >> 2. we need an integration test to make sure that we work with the actual >> >> systems (and possibly several versions) >> >> >> >> This is what Rainer mentioned. >> >> >> >> >> >> >> >> when people are hacking on the code, they may do things that break >> >> processing or variable definitions, and we want a 'known data' test >> that all >> >> we do is a unit test to see if what we send to the remote socket matches >> >> what we expect to send. This doesn't need to talk to a server that >> >> understands the protocol, it just needs a "If we get string 1 reply with >> >> string 2, if we get string 3 reply with string 4" dumb TCP responder. >> >> >> >> Ideally, we would require this sort of test when a module is submitted. >> >> >> >> >> >> >> >> Then as a separate test, we need to talk to Google/Amazon/other VM >> provider >> >> to see about having them donate VM time so that we can setup 'real' >> >> destination systems, and then have a test harness that can clone a >> 'known >> >> good' target, run tests against the clone, then destroy the clone. >> >> >> >> So, there are a few things we would need to do: >> >> >> >> 1. build/find a test harness to run such a setup. >> >> >> >> 2. define what targets we need >> >> >> >> A. what versions of each target need to be tested. >> >> >> >> How many different versions of PostgreSQL are to be supported? >> >> >> >> 3. find 'owners' for each target type. >> >> >> >> This needs to be someone who understands the target to configure the >> >> system, and to respond to issues when we pass the unit test against the >> dumb >> >> responder mentioned above, but fail against the real system. This will >> >> probably involve tinkering with the unit test strings. >> >> >> >> >> >> A. What should we do for target types that require paid licenses? >> >> Ideally the company that owns that target type should be involved to >> provide >> >> us with the license (and some troubleshooting), but in some cases they >> >> aren't going to care. If the vendor isn't going to provide the >> licesnses, >> >> who is going to pay for this? >> >> >> >> 4. Get funding/donations to cover VM time. >> >> >> >> David Lang >> >> >> >> >> >> On Fri, 21 Feb 2014, Rainer Gerhards wrote: >> >> >> >>> Date: Fri, 21 Feb 2014 18:52:29 +0100 >> >>> From: Rainer Gerhards <[email protected]> >> >>> Reply-To: rsyslog-users <[email protected]> >> >>> To: rsyslog-users <[email protected]> >> >>> Subject: Re: [rsyslog] How should the testbench deal with DBs? >> >>> >> >>> >> >>> Let me hijack this thread at least a little... >> >>> >> >>> First: the issue of multiple DB engines and such as real pain in the >> b... >> >>> for me. I'd like to have as many automated tests as possible, but all >> >>> those >> >>> plugins that need "elaborate" setup usually fall short as it is pretty >> >>> hard >> >>> to do (for me and on every dev machine). Remember that for some >> plugins I >> >>> don't even have the slightest idea of how the backend works. And for >> sure >> >>> not even remotely how I could conduct an automated test, which requires >> >>> >> >>> a) getting to a known initial state >> >>> b) shuffeling data at a test run >> >>> c) extracting the results and check if they are ok >> >>> d) probably re-setting things to look untouched again (ok, somewhat >> >>> redundant with a). >> >>> >> >>> Keeping the ES example, I know barely how I can do a, c, and d >> manually so >> >>> that I can conduct manual tests (and after some weeks this even >> requires >> >>> review of notes, as I really don't care/use ES in day-to-day >> operations). >> >>> Doing this in an automated way is way above my head. Now think about >> the >> >>> many other plugins, and those that hopefully will be written. I am not >> >>> that >> >>> super-guru, and I think for many folks it's hard to be really good at >> all >> >>> of these tools (congrats if you are!). >> >>> >> >>> Disabling a test if the feature is not enabled might help an >> end-"user", >> >>> but it doesn't solve the core problem that I don't detect regressions >> >>> right >> >>> in time. >> >>> >> >>> So... now to the hijacking. I have a dream. As a community, I think we >> >>> would be able to create a machine where we could run nightly builds, >> >>> nightly *testbench* runs (on a much more elaborate testbench including >> all >> >>> the backend system), night doc generation and upload ... and so on. >> >>> >> >>> This would need to be a group effort as we would need to drag in all >> the >> >>> experts that could help craft these tests. Also, we would need some >> folks >> >>> who would take care of handling the server itself, I mean the usual >> admin >> >>> things like keeping it secure, creating user accounts for other >> >>> collaborators and so on. I know we have many site reliability engineers >> >>> (and similar positions) on our list, so they could chime in here. Such >> a >> >>> machine would probably also enable me to find bugs quicker. Right now, >> for >> >>> things like ES that I don't work on regularly, the main obstacle to >> start >> >>> a >> >>> bughunt is to setup the env, remember how get around with it and so >> it. If >> >>> there is a ready machine with clear procedure, I could hunt bugs much >> more >> >>> efficient. But quite frankly this is a too large task for us small >> Adiscon >> >>> to handle. We simply have no funding for all that talent required. >> >>> Regarding the platform, I'd assume that a virtual server somewhere in >> the >> >>> cloud could be used (hopefully a sponsored one). And as I am right now >> in >> >>> "whish mode", it would help immensely to have access to maintained >> build >> >>> environments for Solaris, SmartOS, AIX, *BSD and so on, together with >> some >> >>> helping hands I could ask if I don't find may way around easily enough. >> >>> >> >>> OK, that was the dream. Does anybody think we could realize at least a >> bit >> >>> of it in practice? ;) >> >>> >> >>> Thank, >> >>> Rainer >> >>> >> >>> >> >>> On Fri, Feb 21, 2014 at 4:54 PM, Radu Gheorghe >> >>> <[email protected]>wrote: >> >>> >> >>>> Hello increasingly-active-rsyslog-list! >> >>>> >> >>>> I think we need to decide how to move forward with the testbench, >> >>>> whenever people find the time to work on it. The problem came from >> >>>> comments of this issue: >> >>>> https://github.com/rsyslog/rsyslog/issues/10 >> >>>> >> >>>> (check out the comments after the issue was closed) >> >>>> >> >>>> Basically, the general decision point is to see how we should handle >> >>>> rsyslog's integrations with various data stores. Whether that's >> >>>> Elasticsearch, MongoDB, MySQL, etc. >> >>>> >> >>>> So far, dealing with files was done by... wait for it... writing to >> >>>> actual files. This gets more complicated with DBs because you need to >> >>>> have the DB there. Installing it might take time and tons of >> >>>> dependencies. >> >>>> >> >>>> There are a couple of solutions mentioned in the GitHub issue, but I'm >> >>>> throwing a new one after a bit of thinking. Let's assume that the >> >>>> likes of Elasticsearch are already there, and make a list of >> >>>> "requirements" so that one can see what is assumed. >> >>>> >> >>>> The point is, if you don't have ES installed, you probably don't care >> >>>> about ES tests. So all tests about ES could, for instance, try and do >> >>>> a: >> >>>> >> >>>> curl localhost:9200 >> >>>> >> >>>> And if there's an error, or the version is too old, skip the test and >> >>>> print a warning. >> >>>> >> >>>> What do you think? Other suggestions? >> >>>> >> >>>> Best regards, >> >>>> Radu >> >>>> >> >>>> P.S. The point of starting this thread is, as you might see from >> >>>> GitHub, that I've worked on a few omelasticsearch tests, and I'd like >> >>>> to see what we need in order to get them into the main rsyslog >> >>>> testbench. Blindly assuming ES is there is a bit too much, as Rainer >> >>>> pointed out. >> >>>> _______________________________________________ >> >>>> rsyslog mailing list >> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>> http://www.rsyslog.com/professional-services/ >> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >>>> DON'T LIKE THAT. >> >>>> >> >>> _______________________________________________ >> >>> rsyslog mailing list >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>> http://www.rsyslog.com/professional-services/ >> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T >> >>> LIKE THAT. >> >>> >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com/professional-services/ >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad of >> >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T >> >> LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
