2014-11-05 21:55 GMT+01:00 Thomas D. <[email protected]>: > Hi, > > On 2014-11-05 17:33, Rainer Gerhards wrote: > >> Every user should be able to run the test suite. > > > > That's possible today. You just need to setup the environment. I don't > say > > any magic, but in order to run e.g. the mysql tests, you need to have > > > > - mysql server up and running > > - test-specific user credentials and permissions > > - test-specfic databases > > > > if you have that, no problem at all. > > No. I don't have to do that. Calling the test suite from source will > spin- & set-up everything needed, see > http://dev.mysql.com/doc/refman/5.6/en/mysql-test-suite.html > >
The MySQL testbench does not seem to talk to any other backend. So, sure, it does not have any environment dependencies. But how do you test e.g. ommysql with just the rsyslog engine itself? Please don't tell me I should write a MySQL emulator... > >> The test suite must be able to run in a sandbox. > >> > >> That's not possible at the moment. Have a look at mysql, PHP, json-c, > >> salt... this should be possible for rsyslog, too. > > > > I really don't understand that part. If by sandbox you mean that the > > testbench must install everything it needs, then it must setup a complete > > environment, including databases etc. That also means it must query which > > distro it runs on. I've installed VMs for these environments the past > days, > > and 100% scripting this is for sure another big project to do. Also, with > > all that setup, the testbench probably runs 90 minutes or so per > platform - > > and growing. > > Well, you can run the mysql test suite against any mysql server. It is > independent from source. So that's not the best example at all... > Don't you then need root (or similar) credentials to do some very delicate database operations like creating databases and tables? I wouldn't permit this to be done to my production server. > > But my point is: Yes, when you installed all the requirements to build > mysql, the mysql source contains everything to build mysql and to test > what you have actual built. You don't have to create something by hand. > Just call the test suite from source. > > However, I should have looked at "make check" in rsyslog before. It is > not that different from mysql. It spins up its own rsyslog test instance > like mysql is doing... I don't have to create a special environment. > > So yes, seems like there is already something like I had in mind! > > > My short experience with "make check" and rsyslog: > > I did not enabled imdiag. So the last thing I saw was > > > make check-TESTS > > make[2]: Entering directory > '/var/tmp/portage/app-admin/rsyslog-8.9999/work/rsyslog-8.9999/tests' > > make[3]: Entering directory > '/var/tmp/portage/app-admin/rsyslog-8.9999/work/rsyslog-8.9999/tests' > > PASS: manyptcp.sh > > PASS: imptcp_large.sh > > PASS: imptcp_addtlframedelim.sh > > libgcc_s.so.1 must be installed for pthread_cancel to work > > After 20 minutes I found the log file of the current "running" test > which contained the error. > > The test suite should have aborted/failed with that error... if I > wouldn't have stopped the test suite, it would be still running... > Previous versions of autotools did not hide the error message. But you are right, if imdiag is missing the testbench should abort. Also for other required modules (like omstdout). > > Ok; next try with imdiag: > > > [...] > > PASS: pipe_noreader.sh > > PASS: dircreate_dflt.sh > > PASS: dircreate_off.sh > > SKIP: imuxsock_logger_root.sh > > SKIP: imuxsock_traillf_root.sh > > SKIP: imuxsock_ccmiddle_root.sh > > Again I waited 20 minutes... > > According ps it is currently working on udp-msgreduc-vg.sh... > But it seems like the test suite run into an error and again the test > suite didn't notice the problem). > From the test log: > > > [udp-msgreduc-vg.sh]: testing imtcp multiple listeners > > ./diag.sh: line 60: valgrind: command not found > > Yeah, I build with valgrind without valgrind installed ;) > BTW: Are these requirements documented somewhere? > of course not > > Well, finally I installed valgrind and had to re-install glibc with > debugging symbols (another error the test suite logged but did not detect). > > why did you need the debugging symbols? > After valgrind was up and running, the next test which is failing was > > > [rscript_replace_complex.sh]: a more complex test for replace > script-function > > rsyslogd: error during parsing file > ./testsuites/rscript_replace_complex.conf, on or before line 8: syntax > error on token 'set' [try http://www.rsyslog.com/e/2207 ] > > rsyslogd: CONFIG ERROR: could not interpret master config file > './testsuites/rscript_replace_complex.conf'. [try > http://www.rsyslog.com/e/2207 ] > > rsyslogd: run failed with error -2207 (see rsyslog.h or try > http://www.rsyslog.com/e/2207 to learn what that number means) > > ...and again, the test suite doesn't detect that (=no abort/no exit with > an error). The suite would run forever... > > This is something which must be fixed or CI will be a problem. If > there's an error (and the suite logs errors!) it must abort (=exit with > an error). > > I stopped here because the error sounds like a broken test, not a > problem in my environment. > > BTW: A status indicator showing which test is currently running would be > nice, too ;-) > > You are all correct. Patches are happily accepted. But we also need to have a reality check. The system as is works pretty well. External contributions are very low. Those that do notable contributions have the ability to run the testbench themselves. For the others, I find the regressions rather quickly with the existing tooling. So is it really our priority to now make the testbench fully automatic? Again, I would be happy if it were and I am grateful for contributions. But let me be clear. From this thread, it looks like these should be our project priorities: 1. make the testbench fully automatic and self-contained 2. create unit tests 3. create more and better integration tests 4. document 5. fix bugs 6. develop new features Given the fact that I am I and not a big team like at MySQL, I don't believe this is the right order of priorities. In reality, it is me alone doing >95% of all work, and funding for this work is almost solely by Adiscon (and only partially backed by paying customers). > > > Testing all option combinations grows exponentionally, no way around > this. > > If you have 2 binary options, you need 2^2 test, if you have 20, you need > > 2^20 for *full* coverage. If you can do full coverage with fewer tests, > you > > have solved the P=NP problem. It's easy to reduce e.g. the scheduling > > problem to this config testing problem. Poor RSA... ;) > > Just to be clear: About which combinations are we talking? :) > > combination of options. It is possible to use different options, and it is possible to combine different options. As options influence the flow inside rsyslog's code, you need to test all possible option combinations in order to test all execution paths (I think this is called "full path coverage"). Have a look at omfile: http://www.rsyslog.com/doc/master/configuration/modules/omfile.html Full path coverage alone for that module is more than you reasonable can test. Note that I am *not* talking about testing options isolated and hoping that option combinations will work when the isolated cases work. Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
