I wanted to follow up on this.
I've created a feature branch for this and a JIRA issue
https://issues.apache.org/jira/browse/COUCHDB-1762
Overall, I think the worst problem is that the tests really aren't
debuggable in any sane way, and logging is essentially useless for most
things. The only sure way to spot an error most of the time is if it's an
actual CouchDB bug and shows up in the log. I'm not sure how this can ever
be fixed with the current test suite. I'd opt for testing with jasmine, but
that would require not using couchjs for the test runner, so for now, I just
focused on getting random failures under control.
Paul was kind enough to share some code that he wrote recently to deal with
the rampant _restart issues.
https://github.com/davisp/couchdb/commit/0cbf6a9cea01eea599524bcdb77dedb322c7ade4
This is a very sound approach in using a token so you can see if it actually
restarts. The current test suite can result in false positives very easily,
which leads to test failures. I think this is probably the biggest reason
for the random failures. In a previous IRC conversation with Bob (rnewson),
Jan and I think Benoit (sorry if not the case) _restart was deemed something
that should go away. I filed a ticket for it's removal
https://issues.apache.org/jira/browse/COUCHDB-1714, and as Bob points out in
the comments, this is useful for the test suite. I'd argue it's only useful
with Paul's patch adding a token. Otherwise, it's just not reliable at all.
For the branch I created, instead of using _restart, I did some bash magic
with a pipe and stop/start the process through the local run script. This
has the same drawback of not knowing if CouchDB restarted, or we just got a
false positive. To account for this, I put a small delay in the execution of
the lookup, using a new method isRunning to give a little time to stop.
I also changed the suite to run a new couchjs for each test file. I'm not
certain at this point that this is even necessary at all, but I still think
it's safer in case of a crash, since the rest of the suite can continue.
Other changes I made were just timing related in running the test suite for
spinning disks, and a couple bug fixes in individual tests.
The lack of timers makes writing these tests very ugly. I really dislike
this, but so long as the test suite needs couchjs, I don't see a way to
avoid this without implementing our own setInterval method in C.
One last item. I was getting a consistent failure in Centos 6. I tracked
this down to a bug in libcurl. For some reason, after any xhr request that
returns a 416, the very next send() will hang for a long time, then
eventually crash couchjs. The specific version causing the issue is
curl-7.19.7-35.el6 and libcurl-7.19.7-35.el6. I'm not certain if this is
worth reporting in JIRA, but it will certainly cause a test suite failure
consistently in attachment_ranges, but otherwise appears to be fairly
harmless. Maybe this should be documented somewhere?
Wendall
On 03/27/2013 02:05 PM, Wendall Cada wrote:
In 1.3.0, there is a new part of the test suite to run the javascript
tests from the command line. I'm running into various issues on different
hardware/OS configurations. Mostly, tests hanging or timing out and failing.
These are really hard to troubleshoot, as they all pass just fine if run
individually.
What I'm experimenting with today is rewriting how the tests are
implemented to be run one at a time from a loop in bash, versus a loop in
javascript. I think the failures I'm running into are improper
setup/teardown. There may be an issue with rapid delete and adding a db, or
rapidly starting and stopping couchdb, but I think this is not what's
happening in my failures.
The nature of spidermonkey doesn't allow for spawning threads, or
sandboxing, etc, so it's hard looking at the test suite to see how I can
improve running all tests. I think it's far better to have the setup spawn a
new interpreter for each test. Tear down will kill the interpreter.
Wendall
