Hi. Intermittently failing tests seems to be an issue. The way the current tests are scheduled and run (autocrap and make) are not perfect for handling this. I've written a replacement to give us a bit more flexibility wrt to how tests are run.
The below gisted program is a multithreaded python program that run tests. The tests are typically run in parallel (-j n). If a test fails it is scheduled for single threaded execution and these are executed after the rest of the tests are done. This should hopefully reduce the amount of intermittent failures quite with only a small increase in time taken to execute. If we want to gather long term statistics having this program interface with a database of sorts should be reasonably trivial. Another option would be to start giving hints on how the test should be executed within the test itself. By having a header on the test that the testrunner could read one can set up certain parameters. # parallel=0,load<1,timeout=15s,platforms=!netbsd,retries=3 Let me know if I should commit this and start replacement of the current way tests are run. Draft code available at: https://gist.github.com/perbu/3a722df0b168ac8f73bb Syntax: Usage: testrunner <option> [-h] - Show help and exit. [-j <threads>] - paralellize testing (default: 8) [-v] - verbose mode (default is off) [-o <file>] - log to this file (default testrunner.log) [-e <varnishtest>] - set varnishtest path [-l <level>] - set log level (DEBUG,INFO,WARNING,ERROR) [-t <dir>] - look for tests in this directory (default: tests) [-p ] - enable progress [-a <arg>] - pass this argument to varnishtest. Can be used multiple times. -- *Per Buer* CTO | Varnish Software AS Cell: +47 95839117 We Make Websites Fly! www.varnish-software.com [image: Register now] <http://info.varnish-software.com/varnish-summits-autumn-2014-registration>
_______________________________________________ varnish-dev mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
