Hi, On 2014-11-05 17:33, Rainer Gerhards wrote: >> Every user should be able to run the test suite. > > That's possible today. You just need to setup the environment. I don't say > any magic, but in order to run e.g. the mysql tests, you need to have > > - mysql server up and running > - test-specific user credentials and permissions > - test-specfic databases > > if you have that, no problem at all.
No. I don't have to do that. Calling the test suite from source will spin- & set-up everything needed, see http://dev.mysql.com/doc/refman/5.6/en/mysql-test-suite.html >> The test suite must be able to run in a sandbox. >> >> That's not possible at the moment. Have a look at mysql, PHP, json-c, >> salt... this should be possible for rsyslog, too. > > I really don't understand that part. If by sandbox you mean that the > testbench must install everything it needs, then it must setup a complete > environment, including databases etc. That also means it must query which > distro it runs on. I've installed VMs for these environments the past days, > and 100% scripting this is for sure another big project to do. Also, with > all that setup, the testbench probably runs 90 minutes or so per platform - > and growing. Well, you can run the mysql test suite against any mysql server. It is independent from source. So that's not the best example at all... But my point is: Yes, when you installed all the requirements to build mysql, the mysql source contains everything to build mysql and to test what you have actual built. You don't have to create something by hand. Just call the test suite from source. However, I should have looked at "make check" in rsyslog before. It is not that different from mysql. It spins up its own rsyslog test instance like mysql is doing... I don't have to create a special environment. So yes, seems like there is already something like I had in mind! My short experience with "make check" and rsyslog: I did not enabled imdiag. So the last thing I saw was > make check-TESTS > make[2]: Entering directory > '/var/tmp/portage/app-admin/rsyslog-8.9999/work/rsyslog-8.9999/tests' > make[3]: Entering directory > '/var/tmp/portage/app-admin/rsyslog-8.9999/work/rsyslog-8.9999/tests' > PASS: manyptcp.sh > PASS: imptcp_large.sh > PASS: imptcp_addtlframedelim.sh > libgcc_s.so.1 must be installed for pthread_cancel to work After 20 minutes I found the log file of the current "running" test which contained the error. The test suite should have aborted/failed with that error... if I wouldn't have stopped the test suite, it would be still running... Ok; next try with imdiag: > [...] > PASS: pipe_noreader.sh > PASS: dircreate_dflt.sh > PASS: dircreate_off.sh > SKIP: imuxsock_logger_root.sh > SKIP: imuxsock_traillf_root.sh > SKIP: imuxsock_ccmiddle_root.sh Again I waited 20 minutes... According ps it is currently working on udp-msgreduc-vg.sh... But it seems like the test suite run into an error and again the test suite didn't notice the problem). >From the test log: > [udp-msgreduc-vg.sh]: testing imtcp multiple listeners > ./diag.sh: line 60: valgrind: command not found Yeah, I build with valgrind without valgrind installed ;) BTW: Are these requirements documented somewhere? Well, finally I installed valgrind and had to re-install glibc with debugging symbols (another error the test suite logged but did not detect). After valgrind was up and running, the next test which is failing was > [rscript_replace_complex.sh]: a more complex test for replace script-function > rsyslogd: error during parsing file > ./testsuites/rscript_replace_complex.conf, on or before line 8: syntax error > on token 'set' [try http://www.rsyslog.com/e/2207 ] > rsyslogd: CONFIG ERROR: could not interpret master config file > './testsuites/rscript_replace_complex.conf'. [try > http://www.rsyslog.com/e/2207 ] > rsyslogd: run failed with error -2207 (see rsyslog.h or try > http://www.rsyslog.com/e/2207 to learn what that number means) ...and again, the test suite doesn't detect that (=no abort/no exit with an error). The suite would run forever... This is something which must be fixed or CI will be a problem. If there's an error (and the suite logs errors!) it must abort (=exit with an error). I stopped here because the error sounds like a broken test, not a problem in my environment. BTW: A status indicator showing which test is currently running would be nice, too ;-) > Testing all option combinations grows exponentionally, no way around this. > If you have 2 binary options, you need 2^2 test, if you have 20, you need > 2^20 for *full* coverage. If you can do full coverage with fewer tests, you > have solved the P=NP problem. It's easy to reduce e.g. the scheduling > problem to this config testing problem. Poor RSA... ;) Just to be clear: About which combinations are we talking? :) -Thomas _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
