On Thu, Feb 27, 2014 at 1:34 PM, Radu Gheorghe <[email protected]>wrote:
> Join what effort? Of hooking up you guys with Logsene? That would be > my pleasure :D > > Well, what I talk about is setting up the system in such a way that it does nightly builds and test runs. For example, on how to script things so that we get a consistent test environment, run the scripts, provide the results and so on... Quite a bit of work. Very useful work, but probably more than we can do alone. Rainer > On Wed, Feb 26, 2014 at 7:02 PM, Rainer Gerhards > <[email protected]> wrote: > > This sounds *very* interesting. I am currently working on doing at least > > some nightly work on one of our virtual servers. First shot will be at > the > > doc, James is working with me. So far, we wait for the setup ;) I think > > it's best to focus a bit, so do the doc first and then look at the > testing. > > > > It would be even greater if you could join that effort once we are ready. > > > > Rainer > > > > > > On Wed, Feb 26, 2014 at 11:07 AM, Radu Gheorghe <[email protected] > >wrote: > > > >> 2014-02-26 12:01 GMT+02:00 Radu Gheorghe <[email protected]>: > >> > Hello and excuse my late reply, > >> > > >> > I'm willing to help with polishing the ES tests I've already written, > >> > and I guess having new ones plugged in would be easy once we have a > >> > skeleton. > >> > > >> > As for the ES setup itself, I got an idea following David's comment > >> > about VMs. Because I'm working on a log analytics product called > >> > Logsene: > >> > >> Sorry, weird key combination sent the Email outside of my will. Back to > >> Logsene: > >> http://sematext.com/logsene/index.html > >> > >> The trick about it is that it exposes the ES API, so one can use it > >> for tests, proof of concepts, etc. We can have a free account with a > >> free-plan application there that we can use for testing. As long as > >> the build machine has Internet access, it doesn't have to have ES > >> installed, or install it in the first place. But the tests need to > >> have the Logsene application's token stored somewhere. Ideally, the > >> token should be kept private, otherwise anyone could mess up with our > >> tests. > >> > >> Best regards, > >> Radu > >> > >> > > >> > 2014-02-22 21:32 GMT+02:00 David Lang <[email protected]>: > >> >> I think we need two different things. > >> >> > >> >> 1. we need a stand-alone test capability so that people hacking the > >> code can > >> >> make sure that they haven't broken anything. > >> >> > >> >> This is what Radu mentioned. > >> >> > >> >> 2. we need an integration test to make sure that we work with the > actual > >> >> systems (and possibly several versions) > >> >> > >> >> This is what Rainer mentioned. > >> >> > >> >> > >> >> > >> >> when people are hacking on the code, they may do things that break > >> >> processing or variable definitions, and we want a 'known data' test > >> that all > >> >> we do is a unit test to see if what we send to the remote socket > matches > >> >> what we expect to send. This doesn't need to talk to a server that > >> >> understands the protocol, it just needs a "If we get string 1 reply > with > >> >> string 2, if we get string 3 reply with string 4" dumb TCP responder. > >> >> > >> >> Ideally, we would require this sort of test when a module is > submitted. > >> >> > >> >> > >> >> > >> >> Then as a separate test, we need to talk to Google/Amazon/other VM > >> provider > >> >> to see about having them donate VM time so that we can setup 'real' > >> >> destination systems, and then have a test harness that can clone a > >> 'known > >> >> good' target, run tests against the clone, then destroy the clone. > >> >> > >> >> So, there are a few things we would need to do: > >> >> > >> >> 1. build/find a test harness to run such a setup. > >> >> > >> >> 2. define what targets we need > >> >> > >> >> A. what versions of each target need to be tested. > >> >> > >> >> How many different versions of PostgreSQL are to be supported? > >> >> > >> >> 3. find 'owners' for each target type. > >> >> > >> >> This needs to be someone who understands the target to configure > the > >> >> system, and to respond to issues when we pass the unit test against > the > >> dumb > >> >> responder mentioned above, but fail against the real system. This > will > >> >> probably involve tinkering with the unit test strings. > >> >> > >> >> > >> >> A. What should we do for target types that require paid licenses? > >> >> Ideally the company that owns that target type should be involved to > >> provide > >> >> us with the license (and some troubleshooting), but in some cases > they > >> >> aren't going to care. If the vendor isn't going to provide the > >> licesnses, > >> >> who is going to pay for this? > >> >> > >> >> 4. Get funding/donations to cover VM time. > >> >> > >> >> David Lang > >> >> > >> >> > >> >> On Fri, 21 Feb 2014, Rainer Gerhards wrote: > >> >> > >> >>> Date: Fri, 21 Feb 2014 18:52:29 +0100 > >> >>> From: Rainer Gerhards <[email protected]> > >> >>> Reply-To: rsyslog-users <[email protected]> > >> >>> To: rsyslog-users <[email protected]> > >> >>> Subject: Re: [rsyslog] How should the testbench deal with DBs? > >> >>> > >> >>> > >> >>> Let me hijack this thread at least a little... > >> >>> > >> >>> First: the issue of multiple DB engines and such as real pain in the > >> b... > >> >>> for me. I'd like to have as many automated tests as possible, but > all > >> >>> those > >> >>> plugins that need "elaborate" setup usually fall short as it is > pretty > >> >>> hard > >> >>> to do (for me and on every dev machine). Remember that for some > >> plugins I > >> >>> don't even have the slightest idea of how the backend works. And for > >> sure > >> >>> not even remotely how I could conduct an automated test, which > requires > >> >>> > >> >>> a) getting to a known initial state > >> >>> b) shuffeling data at a test run > >> >>> c) extracting the results and check if they are ok > >> >>> d) probably re-setting things to look untouched again (ok, somewhat > >> >>> redundant with a). > >> >>> > >> >>> Keeping the ES example, I know barely how I can do a, c, and d > >> manually so > >> >>> that I can conduct manual tests (and after some weeks this even > >> requires > >> >>> review of notes, as I really don't care/use ES in day-to-day > >> operations). > >> >>> Doing this in an automated way is way above my head. Now think about > >> the > >> >>> many other plugins, and those that hopefully will be written. I am > not > >> >>> that > >> >>> super-guru, and I think for many folks it's hard to be really good > at > >> all > >> >>> of these tools (congrats if you are!). > >> >>> > >> >>> Disabling a test if the feature is not enabled might help an > >> end-"user", > >> >>> but it doesn't solve the core problem that I don't detect > regressions > >> >>> right > >> >>> in time. > >> >>> > >> >>> So... now to the hijacking. I have a dream. As a community, I think > we > >> >>> would be able to create a machine where we could run nightly builds, > >> >>> nightly *testbench* runs (on a much more elaborate testbench > including > >> all > >> >>> the backend system), night doc generation and upload ... and so on. > >> >>> > >> >>> This would need to be a group effort as we would need to drag in all > >> the > >> >>> experts that could help craft these tests. Also, we would need some > >> folks > >> >>> who would take care of handling the server itself, I mean the usual > >> admin > >> >>> things like keeping it secure, creating user accounts for other > >> >>> collaborators and so on. I know we have many site reliability > engineers > >> >>> (and similar positions) on our list, so they could chime in here. > Such > >> a > >> >>> machine would probably also enable me to find bugs quicker. Right > now, > >> for > >> >>> things like ES that I don't work on regularly, the main obstacle to > >> start > >> >>> a > >> >>> bughunt is to setup the env, remember how get around with it and so > >> it. If > >> >>> there is a ready machine with clear procedure, I could hunt bugs > much > >> more > >> >>> efficient. But quite frankly this is a too large task for us small > >> Adiscon > >> >>> to handle. We simply have no funding for all that talent required. > >> >>> Regarding the platform, I'd assume that a virtual server somewhere > in > >> the > >> >>> cloud could be used (hopefully a sponsored one). And as I am right > now > >> in > >> >>> "whish mode", it would help immensely to have access to maintained > >> build > >> >>> environments for Solaris, SmartOS, AIX, *BSD and so on, together > with > >> some > >> >>> helping hands I could ask if I don't find may way around easily > enough. > >> >>> > >> >>> OK, that was the dream. Does anybody think we could realize at > least a > >> bit > >> >>> of it in practice? ;) > >> >>> > >> >>> Thank, > >> >>> Rainer > >> >>> > >> >>> > >> >>> On Fri, Feb 21, 2014 at 4:54 PM, Radu Gheorghe > >> >>> <[email protected]>wrote: > >> >>> > >> >>>> Hello increasingly-active-rsyslog-list! > >> >>>> > >> >>>> I think we need to decide how to move forward with the testbench, > >> >>>> whenever people find the time to work on it. The problem came from > >> >>>> comments of this issue: > >> >>>> https://github.com/rsyslog/rsyslog/issues/10 > >> >>>> > >> >>>> (check out the comments after the issue was closed) > >> >>>> > >> >>>> Basically, the general decision point is to see how we should > handle > >> >>>> rsyslog's integrations with various data stores. Whether that's > >> >>>> Elasticsearch, MongoDB, MySQL, etc. > >> >>>> > >> >>>> So far, dealing with files was done by... wait for it... writing to > >> >>>> actual files. This gets more complicated with DBs because you need > to > >> >>>> have the DB there. Installing it might take time and tons of > >> >>>> dependencies. > >> >>>> > >> >>>> There are a couple of solutions mentioned in the GitHub issue, but > I'm > >> >>>> throwing a new one after a bit of thinking. Let's assume that the > >> >>>> likes of Elasticsearch are already there, and make a list of > >> >>>> "requirements" so that one can see what is assumed. > >> >>>> > >> >>>> The point is, if you don't have ES installed, you probably don't > care > >> >>>> about ES tests. So all tests about ES could, for instance, try and > do > >> >>>> a: > >> >>>> > >> >>>> curl localhost:9200 > >> >>>> > >> >>>> And if there's an error, or the version is too old, skip the test > and > >> >>>> print a warning. > >> >>>> > >> >>>> What do you think? Other suggestions? > >> >>>> > >> >>>> Best regards, > >> >>>> Radu > >> >>>> > >> >>>> P.S. The point of starting this thread is, as you might see from > >> >>>> GitHub, that I've worked on a few omelasticsearch tests, and I'd > like > >> >>>> to see what we need in order to get them into the main rsyslog > >> >>>> testbench. Blindly assuming ES is there is a bit too much, as > Rainer > >> >>>> pointed out. > >> >>>> _______________________________________________ > >> >>>> rsyslog mailing list > >> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >>>> http://www.rsyslog.com/professional-services/ > >> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad > >> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >> >>>> DON'T LIKE THAT. > >> >>>> > >> >>> _______________________________________________ > >> >>> rsyslog mailing list > >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >>> http://www.rsyslog.com/professional-services/ > >> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad > >> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >> DON'T > >> >>> LIKE THAT. > >> >>> > >> >> _______________________________________________ > >> >> rsyslog mailing list > >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> http://www.rsyslog.com/professional-services/ > >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad of > >> >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T > >> >> LIKE THAT. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > > > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
