I think we need two different things.
1. we need a stand-alone test capability so that people hacking the code can
make sure that they haven't broken anything.
This is what Radu mentioned.
2. we need an integration test to make sure that we work with the actual systems
(and possibly several versions)
This is what Rainer mentioned.
when people are hacking on the code, they may do things that break processing or
variable definitions, and we want a 'known data' test that all we do is a unit
test to see if what we send to the remote socket matches what we expect to send.
This doesn't need to talk to a server that understands the protocol, it just
needs a "If we get string 1 reply with string 2, if we get string 3 reply with
string 4" dumb TCP responder.
Ideally, we would require this sort of test when a module is submitted.
Then as a separate test, we need to talk to Google/Amazon/other VM provider to
see about having them donate VM time so that we can setup 'real' destination
systems, and then have a test harness that can clone a 'known good' target, run
tests against the clone, then destroy the clone.
So, there are a few things we would need to do:
1. build/find a test harness to run such a setup.
2. define what targets we need
A. what versions of each target need to be tested.
How many different versions of PostgreSQL are to be supported?
3. find 'owners' for each target type.
This needs to be someone who understands the target to configure the system,
and to respond to issues when we pass the unit test against the dumb responder
mentioned above, but fail against the real system. This will probably involve
tinkering with the unit test strings.
A. What should we do for target types that require paid licenses? Ideally
the company that owns that target type should be involved to provide us with the
license (and some troubleshooting), but in some cases they aren't going to care.
If the vendor isn't going to provide the licesnses, who is going to pay for
this?
4. Get funding/donations to cover VM time.
David Lang
On Fri, 21 Feb 2014, Rainer Gerhards wrote:
Date: Fri, 21 Feb 2014 18:52:29 +0100
From: Rainer Gerhards <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] How should the testbench deal with DBs?
Let me hijack this thread at least a little...
First: the issue of multiple DB engines and such as real pain in the b...
for me. I'd like to have as many automated tests as possible, but all those
plugins that need "elaborate" setup usually fall short as it is pretty hard
to do (for me and on every dev machine). Remember that for some plugins I
don't even have the slightest idea of how the backend works. And for sure
not even remotely how I could conduct an automated test, which requires
a) getting to a known initial state
b) shuffeling data at a test run
c) extracting the results and check if they are ok
d) probably re-setting things to look untouched again (ok, somewhat
redundant with a).
Keeping the ES example, I know barely how I can do a, c, and d manually so
that I can conduct manual tests (and after some weeks this even requires
review of notes, as I really don't care/use ES in day-to-day operations).
Doing this in an automated way is way above my head. Now think about the
many other plugins, and those that hopefully will be written. I am not that
super-guru, and I think for many folks it's hard to be really good at all
of these tools (congrats if you are!).
Disabling a test if the feature is not enabled might help an end-"user",
but it doesn't solve the core problem that I don't detect regressions right
in time.
So... now to the hijacking. I have a dream. As a community, I think we
would be able to create a machine where we could run nightly builds,
nightly *testbench* runs (on a much more elaborate testbench including all
the backend system), night doc generation and upload ... and so on.
This would need to be a group effort as we would need to drag in all the
experts that could help craft these tests. Also, we would need some folks
who would take care of handling the server itself, I mean the usual admin
things like keeping it secure, creating user accounts for other
collaborators and so on. I know we have many site reliability engineers
(and similar positions) on our list, so they could chime in here. Such a
machine would probably also enable me to find bugs quicker. Right now, for
things like ES that I don't work on regularly, the main obstacle to start a
bughunt is to setup the env, remember how get around with it and so it. If
there is a ready machine with clear procedure, I could hunt bugs much more
efficient. But quite frankly this is a too large task for us small Adiscon
to handle. We simply have no funding for all that talent required.
Regarding the platform, I'd assume that a virtual server somewhere in the
cloud could be used (hopefully a sponsored one). And as I am right now in
"whish mode", it would help immensely to have access to maintained build
environments for Solaris, SmartOS, AIX, *BSD and so on, together with some
helping hands I could ask if I don't find may way around easily enough.
OK, that was the dream. Does anybody think we could realize at least a bit
of it in practice? ;)
Thank,
Rainer
On Fri, Feb 21, 2014 at 4:54 PM, Radu Gheorghe <[email protected]>wrote:
Hello increasingly-active-rsyslog-list!
I think we need to decide how to move forward with the testbench,
whenever people find the time to work on it. The problem came from
comments of this issue:
https://github.com/rsyslog/rsyslog/issues/10
(check out the comments after the issue was closed)
Basically, the general decision point is to see how we should handle
rsyslog's integrations with various data stores. Whether that's
Elasticsearch, MongoDB, MySQL, etc.
So far, dealing with files was done by... wait for it... writing to
actual files. This gets more complicated with DBs because you need to
have the DB there. Installing it might take time and tons of
dependencies.
There are a couple of solutions mentioned in the GitHub issue, but I'm
throwing a new one after a bit of thinking. Let's assume that the
likes of Elasticsearch are already there, and make a list of
"requirements" so that one can see what is assumed.
The point is, if you don't have ES installed, you probably don't care
about ES tests. So all tests about ES could, for instance, try and do
a:
curl localhost:9200
And if there's an error, or the version is too old, skip the test and
print a warning.
What do you think? Other suggestions?
Best regards,
Radu
P.S. The point of starting this thread is, as you might see from
GitHub, that I've worked on a few omelasticsearch tests, and I'd like
to see what we need in order to get them into the main rsyslog
testbench. Blindly assuming ES is there is a bit too much, as Rainer
pointed out.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
