On 01/04/2014, at 10:46 PM, Harshavardhana wrote: > As our regression tests have grown bigger and bigger - i have been > observing a pattern of repeated builds which are requested every > patch. > > Here are the list of scenarios that i have observed > > - Testcases pass on local laptop, while fail on Jenkins > - Testcases fail on local laptop, while pass on jenkins > - Testcases fail on both laptop and jenkins but on a second run they pass > - Testcases fail randomly for a totally unrelated patch example
The regression tests really bother me too. :( I've been running the regression tests in several different environments, and some of the tests are very sensitive. By "environment", I'm meaning VMs in my local desktop (in VMware Fusion), and VM's of various cpu/mem/disk configuration in Rackspace. Using CentOS 6.5 in all cases, just to keep consistency until I work things out. Fedora 19/20/etc can pass, but need the -Wall flag removed from build.sh, else the compiler warnings kill the build. These are the completely consistent failures I'm still getting on git master, after having solved most of the others over the last few days: ******************************************************** Test Summary Report ------------------- ./tests/basic/quota.t (Wstat: 0 Tests: 72 Failed: 5) Failed tests: 59-62, 65 ./tests/basic/rpm.t (Wstat: 0 Tests: 4 Failed: 1) Failed test: 4 Parse errors: Bad plan. You planned 5 tests but ran 4. ./tests/bugs/bug-767095.t (Wstat: 0 Tests: 18 Failed: 1) Failed test: 13 Parse errors: Bad plan. You planned 19 tests but ran 18. ./tests/bugs/bug-861542.t (Wstat: 0 Tests: 13 Failed: 1) Failed test: 10 ./tests/bugs/bug-963678.t (Wstat: 0 Tests: 16 Failed: 8) Failed tests: 5-11, 13 Parse errors: Bad plan. You planned 18 tests but ran 16. ******************************************************** Test 65 of quota.t is a known bug (3.5 blocker). Varun is aware of it: https://bugzilla.redhat.com/show_bug.cgi?id=1077159 The rpm.t one is likely something to do with the environment, (eg no tty), as when I run it manually it's fine. Haven't gotten up to investigating it properly yet, but will soon. The most difficult thing so far has been the lack of logging for stdout and stderr. Everything useful seems to be redirected to /dev/null instead of $TESTNUM.log (or $TESTNUM.stdout & $TESTNUM.stderr). With the exception of rpm.t that is, which has an optional DEBUG=1 flag. Also non-great is not being able to effectively debug bash scripts. eg set breakpoint, step, step, check variable, aha! problem found, (etc). But, that's not as critical as the lack of logging. I'm not seeing as much random/inconsistent failure with the tests once the VM they're is sized ok (enough memory mainly), but it still does happen. Don't suppose you're really good with bash scripting, and know it well enough to fix the logging problem? I had a go the other day, and it was just too complex for me (ended up writing a new basic framework in Python). Regards and best wishes, Justin Clift -- Open Source and Standards @ Red Hat twitter.com/realjustinclift _______________________________________________ Gluster-devel mailing list Gluster-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/gluster-devel
