We have been looking to reuse the CouchDB javascript tests for PouchDB and extracted them into a seperate npm package
https://npmjs.org/package/couchdb-harness The idea was that we could reused the same test suite and throw ours away however it hit a few problems, mostly that the test suite requires a sync api (which caused most of the problems with xhr in the browser couch tests), I spent a bunch of time converting the test suites to async, they dont map 1-1 because the couch suite is quite imperative and not really split into self contained unit tests. But already have 1000+ tests that test CouchDB, work on the command line and reliably in the browser, I dont think they are suitable for couch right now, but would be happy to do the work so we could share a test suite https://github.com/daleharvey/pouchdb/tree/master/tests On 8 April 2013 15:26, Paul Davis <paul.joseph.da...@gmail.com> wrote: > The first bit I'd like to say is that the use of couchjs was just a > stop gap measure to get the test suite out of the browser. We used to > have to deal with so many browser issues it was just a terrible mess. > The issue with couchjs is much as you've seen that its not a very full > environment for writing tests. So just to be clear that the only real > thing tying us to that as a test platform is that we have a large > amount of JS written already so either we need to make the couchjs > better, use node, or translate tests to something that has a more > useful environment. > > I've been noodling over whether we might be better off to just start > translating everything to Python or something. I've seen suggestions > for Erlang but I personally think Erlang is a terrible language for > writing tests like this (specifically, the code to test ratio is > ungood). If we had something like Python to hack on then I was also > thinking of writing a library function that would start CouchDB as a > slave process which then would remove the need to have the _restart > handler because you could just kill -9 the subprocess and restart it > with maybe a wait for when things boot again. > > I reviewed your feature branch the other day and I'm +1 for pushing > that to master. > > Awesome work, Wendall. > > On Fri, Apr 5, 2013 at 6:57 PM, Wendall Cada <wenda...@apache.org> wrote: > > I wanted to follow up on this. > > > > I've created a feature branch for this and a JIRA issue > > https://issues.apache.org/jira/browse/COUCHDB-1762 > > > > Overall, I think the worst problem is that the tests really aren't > > debuggable in any sane way, and logging is essentially useless for most > > things. The only sure way to spot an error most of the time is if it's an > > actual CouchDB bug and shows up in the log. I'm not sure how this can > ever > > be fixed with the current test suite. I'd opt for testing with jasmine, > but > > that would require not using couchjs for the test runner, so for now, I > just > > focused on getting random failures under control. > > > > Paul was kind enough to share some code that he wrote recently to deal > with > > the rampant _restart issues. > > > https://github.com/davisp/couchdb/commit/0cbf6a9cea01eea599524bcdb77dedb322c7ade4 > > This is a very sound approach in using a token so you can see if it > actually > > restarts. The current test suite can result in false positives very > easily, > > which leads to test failures. I think this is probably the biggest reason > > for the random failures. In a previous IRC conversation with Bob > (rnewson), > > Jan and I think Benoit (sorry if not the case) _restart was deemed > something > > that should go away. I filed a ticket for it's removal > > https://issues.apache.org/jira/browse/COUCHDB-1714, and as Bob points > out in > > the comments, this is useful for the test suite. I'd argue it's only > useful > > with Paul's patch adding a token. Otherwise, it's just not reliable at > all. > > > > For the branch I created, instead of using _restart, I did some bash > magic > > with a pipe and stop/start the process through the local run script. This > > has the same drawback of not knowing if CouchDB restarted, or we just > got a > > false positive. To account for this, I put a small delay in the > execution of > > the lookup, using a new method isRunning to give a little time to stop. > > > > I also changed the suite to run a new couchjs for each test file. I'm not > > certain at this point that this is even necessary at all, but I still > think > > it's safer in case of a crash, since the rest of the suite can continue. > > > > Other changes I made were just timing related in running the test suite > for > > spinning disks, and a couple bug fixes in individual tests. > > > > The lack of timers makes writing these tests very ugly. I really dislike > > this, but so long as the test suite needs couchjs, I don't see a way to > > avoid this without implementing our own setInterval method in C. > > > > One last item. I was getting a consistent failure in Centos 6. I tracked > > this down to a bug in libcurl. For some reason, after any xhr request > that > > returns a 416, the very next send() will hang for a long time, then > > eventually crash couchjs. The specific version causing the issue is > > curl-7.19.7-35.el6 and libcurl-7.19.7-35.el6. I'm not certain if this is > > worth reporting in JIRA, but it will certainly cause a test suite failure > > consistently in attachment_ranges, but otherwise appears to be fairly > > harmless. Maybe this should be documented somewhere? > > > > Wendall > > > > > > On 03/27/2013 02:05 PM, Wendall Cada wrote: > >> > >> In 1.3.0, there is a new part of the test suite to run the javascript > >> tests from the command line. I'm running into various issues on > different > >> hardware/OS configurations. Mostly, tests hanging or timing out and > failing. > >> These are really hard to troubleshoot, as they all pass just fine if run > >> individually. > >> > >> What I'm experimenting with today is rewriting how the tests are > >> implemented to be run one at a time from a loop in bash, versus a loop > in > >> javascript. I think the failures I'm running into are improper > >> setup/teardown. There may be an issue with rapid delete and adding a > db, or > >> rapidly starting and stopping couchdb, but I think this is not what's > >> happening in my failures. > >> > >> The nature of spidermonkey doesn't allow for spawning threads, or > >> sandboxing, etc, so it's hard looking at the test suite to see how I can > >> improve running all tests. I think it's far better to have the setup > spawn a > >> new interpreter for each test. Tear down will kill the interpreter. > >> > >> Wendall > > > > >
