On Sun, Aug 21, 2011 at 9:34 PM, Konstantin Boudnik <c...@apache.org> wrote: > System tests (Herriot controlled) tests were a part of nightly testing of > every > build for at least 2 of .2xx release. I really can not comment on .203 and > after.
Owen - are you running the system tests on the 20x release candidates? Do we know if the 20x release pass the system tests? > A normal procedure was to build a normal bits and run the tests; build > instrumented bits, deploy them to a 10 nodes cluster, and run system tests. > The current state of the code is that system tests require source code > workspace to be executed from. I have done some initial work to do workspace > independent testing but I don't know if it has been included to the public > releases of .203+ - I haven't really checked. > > At any rate, running system tests are an easy task and the wiki page is > explaining how to do it. Running the system tests is actually not easy, those wiki instructions are out of date, require all kinds of manual steps, and some of the tests fail when just run from a local build (ie they require 3 DNs so you have to setup a cluster). > Assembling an instrumented cluster on the other hand > requires certain knowledge and release process and bits production. > Instrumented cluster isn't fault-injected - it is just instrumented ;) Yes, it > contains a few extra helper API calls in a few classes, which exactly makes > them a way more useful for the testing purpose. Without those a number of > testing scenarios would be impossible to implement as I have explained it on > many occasions. Could you point me to a thread that covers the few extra helper API calls that are injected? I can't see what API would both be necessary for a system test and also not able be included in the product itself. If you're system testing an instrumented build than you're not system testing the product used by users. > > For the regular runs of system test Roman and I have created a regular > deployment of 0.22 cluster builds under Apache Hudson control a few months > ago. I don't know what's going on with this testing after recent troubles with > the build machines. How hard would it be to copy your 22 system test Jenkins job to adapt it to use a 20x build? Seems like the test bits should mostly be the same. Thanks, Eli
