Started with python dev/run -n 1 --with-admin-party-please &
and export COUCH_HOST='http://127.0.0.1:15984' On 12 October 2015 at 13:26, Jan Lehnardt <[email protected]> wrote: > > > On 12 Oct 2015, at 13:15, Dale Harvey <[email protected]> wrote: > > > >> - instead of fixed test db names like `test_suite_db` that are used in > > almost every test, > >> we now generate a random database name for each test. The JS tests > > expected db > >> deletion to be synchronous, and that’s no longer the case in 2.0. > Instead > > of > >> error-prone polling to wait for db deletion, we just use new db for > every > > test. > >> An added benefit now is that (some) tests can run in parallel. > > > > Little confused by this, PouchDB uses consistent names in its test suite > > (we had random > > ones for a while and moved away), we very much depend on the database > being > > deleted > > on the request returning and dont seem to have any problems with it > against > > master. > > Does the PouchDB test suite runs against the 5986 port, or with an n=1 > cluster? > This is with the default n=3 cluster against 5986. > > Best > Jan > -- > > > > > On 12 October 2015 at 12:21, Jan Lehnardt <[email protected]> wrote: > > > >> > >>> On 11 Oct 2015, at 18:53, Sebastian Rothbucher < > >> [email protected]> wrote: > >>> > >>> Hi Jan, > >>> > >>> I transferred some of my findings to your branch - pls. find a PR for > the > >>> first few tests submitted 2 your branch. I put in TODOs where either > >> there > >>> is most probably a bug (filed COUCHDB-2852 and COUCHDB-2853) or we have > >> to > >>> do some more work (like making config available). > >> > >> Excellent, thanks! :) > >> > >> > >>> Will from the Pouch team had the great idea of using -n 1 to mitigate > >>> Heisenbugs due to distribution. I adopted it and it did work quite > well. > >> It > >>> might make sense to put this into the dev/run call as a general rule, > >> esp. > >>> when checking for 201 (VS 202) status codes. Probably you know more on > >>> that. > >> > >> We should carefully evaluate this :) > >> > >> > >>> Anyways: I think we have to keep using _config on the local port as > many > >>> tests (like auth tests) depend on changed configurations. BTW: w/ -n 1 > >> it's > >>> only one call, but calling 3 _config(s) migt also work. > >>> Hopefully (I just arrived at some test w/ that recently) we can spare > >>> ourselves the _compact thing. I think we'd have to compact every single > >>> shard via local port, right? for the first tests, we can go w/out as it > >>> seems. Maybe we should stick to that for a while. > >> > >> My concern with this is that as far as I care, the local port is > irrelevant > >> to how CouchDB works. Of course it isn’t 100% irrelevant, but the > cluster > >> port is our publicised API for 2.0 and it’s the one that replaces the > 1.x > >> API for people upgrading. So having any tests that run on the local port > >> of 2.0 don’t help us in any way (that’s not to say it still useful to > have > >> tests against them for the moment). > >> > >> Many of the _config-dependent tests substitute out the _users or > >> _replicator > >> databases, maybe we can find a workaround for those by just using the > stock > >> databases. > >> > >> For the other tests, maybe we split them out into their own “test > suites” > >> that start their own test cluster in the configuration they need and > shut > >> it down afterwards. > >> > >> Best > >> Jan > >> -- > >> > >> > >>> > >>> Looking fw 2 your thoughts > >>> > >>> Best > >>> Sebastian > >>> > >>> P.S:: Maybe we can adopt some stuff from > >>> > >> > https://github.com/apache/couchdb/compare/master...sebastianrothbucher:clustertest > >>> > >>> P.P.S.: Will keep working on it, just not in the next few days, > >>> unfortunately ;-( > >>> > >>> On Sun, Oct 11, 2015 at 12:00 PM, Jan Lehnardt <[email protected]> wrote: > >>> > >>>> (I need your help deciding what to do with a bunch of tests, jump to > >>>> “Discussion” below if you don’t care about the details before.) > >>>> > >>>> * * * > >>>> > >>>> Hi all, > >>>> > >>>> I spent some more time getting the 1.x JavaScript tests running on the > >>>> cluster port of 2.0. WIP here: > >> https://github.com/apache/couchdb/pull/354 > >>>> — I’ve gotten a lot further, but there is still a lot to do and > everyone > >>>> here can help, I’ve posted instructions on how to do so further down. > No > >>>> Erlang knowledge needed, this is ideal for folks who want to start > with > >>>> contributing to CouchDB. > >>>> > >>>> > >>>> Notable changes: > >>>> > >>>> - instead of fixed test db names like `test_suite_db` that are used in > >>>> almost every test, we now generate a random database name for each > test. > >>>> The JS tests expected db deletion to be synchronous, and that’s no > >> longer > >>>> the case in 2.0. Instead of error-prone polling to wait for db > >> deletion, we > >>>> just use new db for every test. An added benefit now is that (some) > >> tests > >>>> can run in parallel. > >>>> > >>>> - tests now clean up after themselves (incomplete): tests previously > >> only > >>>> collectively used three or four databases, and deleted and re-created > >> them > >>>> before each test was run. Now with the change of creating a new > database > >>>> per test, we also need to clean up databases at the end of a test, > >> because > >>>> otherwise we a) leave a lot of databases around and b) CouchDB runs > out > >> of > >>>> file descriptors during the test run. The cleanup isn’t 100% done yet, > >> so > >>>> we should check for stray databases after JS test runs until we’ve got > >> it > >>>> cleaned. One particular aspect here is when a test fails, it never > >> reaches > >>>> the cleanup code, so we might have to wrap everything into a try/catch > >>>> block. > >>>> > >>>> - I disabled all tests that still fail and made them so they print the > >>>> reason why they are failing, so that fixing them should be a lot > easier > >>>> now. Some of them print just “TODO”, their error cause is yet to be > >>>> determined. > >>>> > >>>> - couchdb.query() which previously executed temporary views, now under > >> the > >>>> hood creates a design doc with a random name and queries that view. > This > >>>> fixes all tests relying on temp views without having to change them > all > >>>> individually. This is because 2.0 doesn’t have temp views anymore > >> (which is > >>>> a good thing). > >>>> > >>>> - restartServer() is no longer used (mostly). > >>>> > >>>> I already filed a few bugs that showed things missing in the cluster > API > >>>> that we’d expect to be there: > >>>> > >>>> - https://issues.apache.org/jira/browse/COUCHDB-2849 > >>>> - https://issues.apache.org/jira/browse/COUCHDB-2850 > >>>> - https://issues.apache.org/jira/browse/COUCHDB-2851 > >>>> > >>>> Based on this I am ever more convinced we should get this test suite > >> work > >>>> against the 2.0 clustered API, because we’ll be finding some more > >>>> inconsistencies that we should fix before we invite beta testers. > >>>> > >>>> > >>>> ## Discussion: > >>>> > >>>> We have a whole number of tests that use the test-internal function > >>>> run_on_modified_server. They use the _config API to test CouchDB in > >>>> different configurations from the default one. All of these tests now > >> throw > >>>> an error that _config isn’t available on the clustered API. How should > >> we > >>>> handle them? > >>>> > >>>> The tests are: > >>>> > >>>> - erlang_views.js > >>>> - jsonp.js > >>>> - oauth_users_db.js > >>>> - oauth_users_db.js > >>>> - proxyauth.js > >>>> - reader_acl.js > >>>> - rewrite.js > >>>> - security_validation.js > >>>> - show_documents.js > >>>> - users_db.js > >>>> - view_errors.js > >>>> > >>>> > >>>> And a few tests aside from the proper compaction tests call _compact > >> which > >>>> is also not available on the cluster. What to do with them? > >>>> > >>>> - recreate_doc.js > >>>> - and a few more that are currently disabled for other reasons, sorry > I > >>>> don’t have a full list at this point. > >>>> > >>>> * * * > >>>> > >>>> > >>>> Bottom line: It’d be great if other folks here could join in the fun. > No > >>>> Erlang needed, this is purely JavaScript and CouchDB HTTP API-work :) > >>>> > >>>> > >>>> ## How to help: > >>>> > >>>> 1. Check out https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip > >>>> 2. run `make clean && ./configure -c --disable-docs --disable-fauxton > && > >>>> make` > >>>> 3. `make javascript` runs the whole test suite > >>>> > >>>> To run the whole test suite without waiting for rebar to confirm that > no > >>>> Erlang code needs to be compiled, run `./test/javascript/run` for the > >> whole > >>>> test suite, or `./test/javascript/run all_docs` to run an individual > >> test > >>>> (all_docs runs test/javascript/tests/all_docs.js, other file names > match > >>>> accordingly). > >>>> > >>>> Feel free to send me PRs against the branch on > >>>> https://github.com/janl/couchdb/tree/js-tests-on-2.0-wip, I’ll merge > >>>> them, so the PR on apache/couchdb gets updated. > >>>> > >>>> I’m looking forward to your contributions! :) > >>>> > >>>> If you have any questions or have trouble getting started or just > would > >>>> like some hand-holding, let me know! > >>>> > >>>> > >>>> Best > >>>> Jan > >>>> -- > >>>> > >>>> > >>>> > >>>> > >> > >> -- > >> Professional Support for Apache CouchDB: > >> http://www.neighbourhood.ie/couchdb-support/ > >> > >> > > -- > Professional Support for Apache CouchDB: > http://www.neighbourhood.ie/couchdb-support/ > >
