I just updated the README in the Elixir suite to have a todo list of tests to port, in case folks are looking for tests: https://github.com/apache/couchdb/blob/elixir-suite/test/elixir/README.md
-Russell On Fri, Dec 15, 2017 at 9:46 AM Paul Davis <paul.joseph.da...@gmail.com> wrote: > For `make check` it should be fairly straightforward to map the > current approach to it. I could probably knock that out fairly quickly > if you want me to give it a whirl. > > On Fri, Dec 15, 2017 at 11:42 AM, Russell Branca <chewbra...@apache.org> > wrote: > > Yeah just to reiterate what Paul said, the Elixir dev experience is > really > > nice and easy to get rolling with. I had no prior actual experience with > > Elixir and I was able to get things rolling in a few hours. > > > > RE Ben's question about diving in: please do! Just grab one of the > unported > > js suites and goto town. I've just been cherry-pick'ing things out of > > Paul's branch and we can continue to do the same until we get this more > > locked down. My goal with the porting is to keep chugging along and just > > get it knocked out, as I really don't think it will be overly onerous to > do > > so. And if anyone else wants to jump in, there's still a fair number of > > tests to port, just take your pick. > > > > One other thing that needs work is figuring out how to hook all this into > > "make check" and what not. I've mostly ignored that as this just points > at > > a CouchDB instance and can be run directly, but we'll need to sort that > out > > at some point. > > > > > > -Russell > > > > On Fri, Dec 15, 2017 at 9:03 AM Paul Davis <paul.joseph.da...@gmail.com> > > wrote: > > > >> Hello everybody! > >> > >> I figured I should probably go ahead and chime in seeing as I've also > >> been playing around porting some of the tests in my free time between > >> ops shifts the last couple weeks. > >> > >> My first impression was that it was ridiculously easy to get involved. > >> On OS X at least, `brew install elixir` was enough to get a working > >> elixir installed (however, if you use kerl or erln8 you'll want have > >> to build an Erlang 20.x VM to use the brew package). I went from not > >> having Elixir installed to a full port of uuids.js with the config tag > >> logic written in about two hours one night. So far the Elixir docs and > >> seem very well written and put together. I'd say the worst part of > >> Elixir so far is that knowing Erlang I find myself searching for "How > >> do I do this Erlang thing in Elixir?" Which isn't as bad as it sounds. > >> The Elixir libraries have certainly had a considerable amount of > >> thought put into them to make them easy to use and remember. I find it > >> to be a lot like my experience when learning Python in that I may have > >> to Google once and then its muscle memory. As opposed to Erlang's > >> library where I'm constantly reading the lists manpage to remember > >> argument orderings and whether I want search or find versions etc. > >> > >> Which I guess is a long way of saying I'm rather liking the Elixir > >> development experience so far. > >> > >> That said, I'm currently about half way through porting replication.js > >> tests to Elixir. For the most part its fairly straightforward. My > >> current approach as we've done for the other modules is to do a direct > >> port. Once that's finished we'll want to break up that huge module > >> into a series of modules that share a lot of the utility functions. > >> One of the nice things about moving to Elixir is that its got a full > >> on development story rather than our current couchjs approach that > >> prevents sharing code easily between subsets of tests. > >> > >> For Ben's question on diving in, I'd do just that. I'd say leave a > >> note here about which module(s)? you're going to port so that we're > >> not duplicating efforts and then its basically just a matter of > >> getting Elixir installed. For that, here's a quick rundown on how I > >> got that working: > >> > >> $ brew update > >> $ brew install elixir > >> $ # wait for all the things... > >> $ iex # which fails cause I have an Erlang VM older than 20.0 as a > default > >> $ erln8 --fetch > >> $ erln8 --build --tag=OTP-20.1.6 --id=20.1.6 > >> $ # wait while erln8 does its thing > >> $ git clone https://github.com/apache/couchdb > >> $ cd couchdb > >> $ ./configure --disable-docs --disable-fauxton --with-curl > >> $ make > >> $ git checkout -b elixir-suite-davisp origin/elixir-suite # but use > >> your own name > >> $ cd elixir_suite # Russel promises to move this to test/elixir > >> eventually... :) > >> $ mix deps.get > >> $ # For the moment, in another terminal, run ./dev/run -a adm:pass > >> $ mix test --trace > >> $ # For development you can also do this: > >> $ mix test --trace test/module_i_am_working_on.exs > >> > >> For the time being, anyone that does any porting work, I'd just let > >> Russel know and he can pull the changes into the main elixir-suite > >> branch. For the initial work it might get a bit messy but we can > >> always clean up after the fact if we decide this is a direction we'd > >> like to go for real. To that end, I'd also make sure that we do a > >> single .js -> .exs port per commit to try and make any future cleanup > >> work easier. > >> > >> Also, even if people don't feel like doing any actual porting work I'd > >> still be interested in hearing what its like for people to just run > >> through their platform equivalent of the above steps. And even just > >> initial impressions on toying around with Elixir. My only experience > >> with Elixir prior to this was reading through their quick > >> start/tutorial pages a couple of times to get a feeling for the syntax > >> but hadn't actually even typed it into an editor till last week. > >> > >> And that's all I've got for now. > >> > >> On Thu, Dec 14, 2017 at 11:57 PM, Benjamin Anderson > >> <banjie...@apache.org> wrote: > >> > Slick! This seems like it's coming together really nicely. Can't argue > >> > with commits like "Prefer ?w=3 over hacky sleeps"[1] in any case. > >> > > >> >> I hope others have similar opinions after diving in! > >> > > >> > How should one dive in? Are you looking for others to help out with > >> > the ports, or just thinking aspirationally about future regular > >> > contributions to the test suite? > >> > > >> > -- > >> > b > >> > > >> > [1]: > >> > https://github.com/apache/couchdb/commit/5bce2d98a298c25b77d8dcda19deeedb494cc289 > >> > > >> > On Thu, Dec 14, 2017 at 5:03 PM, Russell Branca < > chewbra...@apache.org> > >> wrote: > >> >> Howdy folks! > >> >> > >> >> The testing of CouchDB is something that has seen focus and > improvements > >> >> for the last several years, for instance migrating the etap suite to > >> eunit, > >> >> and updating the JS suite to run against clusters in 2.x. There's > still > >> >> improvements to be made, and that was one of the topics of the > CouchDB > >> dev > >> >> summit early in the year [1]. > >> >> > >> >> Before we go further, I want to clarify some nomenclature. I'm by no > >> means > >> >> going to try and define unit testing vs integration testing vs > quantum > >> >> phase shift testing, but instead I want to focus on the distinction > of > >> >> where the testing takes place. Fundamentally, we have two places we > test > >> >> CouchDB: 1) at the Erlang VM level where we conduct assertions > against > >> >> module functions or process states; 2) at the HTTP level where we > test > >> the > >> >> behavior of CouchDB at the user level API. This post focuses > entirely on > >> >> the latter; that's not to say the former doesn't also merit > attention, > >> just > >> >> that the two are different enough that we can focus on them in > >> isolation. > >> >> > >> >> So with that, let's chat about the current HTTP test suite in > CouchDB. > >> This > >> >> is the "JS suite" I referred to above, which is a custom built test > >> suite > >> >> written in Javascript and executed in the aging SpiderMonkey. The JS > >> suite > >> >> has put in work for years, but it's showing it's age, and is a bit > >> awkward > >> >> to work with and improve. However, I think the biggest issue with > the JS > >> >> suite is that it's utilized far less than it should be, and folks > seem > >> to > >> >> avoid extending it or adding additional tests to it. There's been > >> >> discussion for years about replacing said suite, but the discussions > >> >> invariably got blocked on the bike shed of whether to rewrite the > suite > >> in > >> >> Javascript or Python. This thread provides a third option, with code! > >> >> > >> >> I started hacking on a replacement for the JS suite, this time > written > >> in > >> >> Elixir. Overall I'm quite impressed with how it's come along, and > have > >> some > >> >> good examples to show. This is basically an Elixir app that has an > HTTP > >> >> client and then runs a series of tests that conduct tests against the > >> >> CouchDB HTTP API and make assertions therein. > >> >> > >> >> You can find the current code in [2], and a comparison of the > changes in > >> >> [3]. The core HTTP client is only a handful of lines of codes and > works > >> >> quite well [4]. The utility functions used across all tests are > located > >> in > >> >> [5], and the tests themselves are in [6]. The existing test modules > >> have a > >> >> 1:1 correspondence with the associated JS suite test modules, and in > >> >> general are as direct of a port as possible. > >> >> > >> >> The test modules ported in their entirety or most of the way are: > >> >> > >> >> * all_docs.js > >> >> * basics.js > >> >> * config.js > >> >> * reduce.js > >> >> * rewrite.js > >> >> * uuids.js > >> >> * view_collation.js > >> >> > >> >> Paul has dove in and is responsible for a few of those test modules > and > >> >> he's almost completed porting the replication.js suite as well. We > >> started > >> >> with the hard ones first, so for the most part the rest of the ports > >> should > >> >> be fairly smooth sailing. > >> >> > >> >> Here's an example of a very basic test: > >> >> > >> >> ```erlang > >> >> defmodule WelcomeTest do > >> >> use CouchTestCase > >> >> > >> >> test "Welcome endpoint" do > >> >> assert Couch.get("/").body["couchdb"] == "Welcome", "Should say > >> welcome" > >> >> end > >> >> > >> >> end > >> >> > >> >> ``` > >> >> > >> >> > >> >> As you can see, the `Couch` client is very simple HTTP client with > >> >> easy HTTP verb based methods. Let's look at a more complicated test > >> >> for asserting we can create documents in a database: > >> >> > >> >> > >> >> ```erlang > >> >> > >> >> @tag :with_db > >> >> test "Create a document and save it to the database", context do > >> >> resp = Couch.post("/#{context[:db_name]}", [body: %{:_id => "0", > >> >> :a => 1, :b => 1}]) > >> >> assert resp.status_code == 201, "Should be 201 created" > >> >> assert resp.body["id"], "Id should be present" > >> >> assert resp.body["rev"], "Rev should be present" > >> >> > >> >> resp2 = Couch.get("/#{context[:db_name]}/#{resp.body["id"]}") > >> >> assert resp2.body["_id"] == resp.body["id"], "Ids should match" > >> >> assert resp2.body["_rev"] == resp.body["rev"], "Revs should > match" > >> >> end > >> >> > >> >> ``` > >> >> > >> >> > >> >> This is fairly straightforward code to POST a new doc, make > assertions > >> >> on the response, and then fetch the doc to make sure everything > >> >> matches up. What I really wanted to highlight here is the `@tag > >> >> :with_db` decorator. We can easily add custom "tags" to the tests to > >> >> simplify setup and teardown. That `:with_db` tag does two things, it > >> >> dynamically generates a random database name, and then takes care of > >> >> setup/teardown for creating and deleting said database for that > >> >> particular test. This is really useful and has been very nice to work > >> >> with so far. We also have tag functionality in place for executing a > >> >> test with a particular set of config options: > >> >> > >> >> > >> >> ```erlang > >> >> > >> >> @tag config: [ > >> >> {"uuids", "algorithm", "utc_random"} > >> >> ] > >> >> test "utc_random uuids are roughly random" do > >> >> resp = Couch.get("/_uuids", query: %{:count => 1000}) > >> >> assert resp.status_code == 200 > >> >> uuids = resp.body["uuids"] > >> >> > >> >> assert String.length(Enum.at(uuids, 1)) == 32 > >> >> > >> >> # Assert no collisions > >> >> assert length(Enum.uniq(uuids)) == length(uuids) > >> >> > >> >> # Assert rough ordering of UUIDs > >> >> u1 = String.slice(Enum.at(uuids, 1), 0..13) > >> >> u2 = String.slice(Enum.at(uuids, -1), 0..13) > >> >> assert u1 < u2 > >> >> end > >> >> ``` > >> >> > >> >> > >> >> The tag system really simplifies a lot of the standard auxiliary > >> >> actions needed to conduct tests. > >> >> > >> >> > >> >> To test out the suite, you'll need to spin up the dev server in one > >> window with: > >> >> > >> >> > >> >> ``` > >> >> > >> >> ./dev/run --admin=adm:pass > >> >> > >> >> ``` > >> >> > >> >> > >> >> and then in another window go into the relevant CouchDB src directory > >> and run: > >> >> > >> >> > >> >> ``` > >> >> > >> >> cd ~/src/couchdb/elixir_suite/ > >> >> > >> >> mix deps.get > >> >> > >> >> mix test --trace > >> >> > >> >> ``` > >> >> > >> >> > >> >> The `--trace` flag makes the nice line item output per test, which I > >> >> greatly prefer over a slew of periods. You can run an individual test > >> >> with `mix test --trace tests/basics_test.exs`. I've pasted the output > >> >> from running the basics suite at the bottom of this email so you can > >> >> see what the real output looks like. > >> >> > >> >> > >> >> Overall I'm quite impressed with the toolkit we've been able to put > >> >> together in a short amount of time, and I propose we migrate fully to > >> >> this test suite by porting all remaining JS suite tests and then > >> >> removing the JS suite entirely. Given we've already ported most of > the > >> >> "hard suites", I think a full port is reasonable to do and just > >> >> requires some leg work. Again, I'm impressed with how simple the > >> >> tooling here is and how quickly we've been able to run with things, > >> >> turns out the Elixir dev experience is actually quite nice! I hope > >> >> others have similar opinions after diving in! Let me know what you > >> >> think. > >> >> > >> >> > >> >> > >> >> -Russell > >> >> > >> >> > >> >> > >> >> [1] https://github.com/janl/couchdb-next/issues/39 > >> >> [2] https://github.com/apache/couchdb/tree/elixir-suite > >> >> [3] https://github.com/apache/couchdb/compare/elixir-suite > >> >> [4] > >> >> > >> > https://github.com/apache/couchdb/blob/elixir-suite/elixir_suite/lib/couch.ex > >> >> [5] > >> >> > >> > https://github.com/apache/couchdb/blob/elixir-suite/elixir_suite/test/test_helper.exs > >> >> [6] > >> https://github.com/apache/couchdb/tree/elixir-suite/elixir_suite/test > >> >> > >> >> > >> >> vagrant@contrib-jessie:~/src/couchdb/elixir_suite$ mix test --trace > >> >> test/basics_test.exs > >> >> Excluding tags: [pending: true] > >> >> > >> >> BasicsTest > >> >> * test Session contains adm context (66.8ms) > >> >> * test Creating a new DB with slashes should return Location header > >> >> (COUCHDB-411) (85.8ms) > >> >> * test oops, the doc id got lost in code nirwana (82.1ms) > >> >> * test Welcome endpoint (7.6ms) > >> >> * test POST doc with an _id field isn't overwritten by uuid > (102.7ms) > >> >> * test On restart, a request for creating an already existing db > can > >> >> not override (skipped) > >> >> * test Creating a new DB should return location header (118.7ms) > >> >> * test _bulk_docs POST error when body not an object (95.0ms) > >> >> * test Empty database should have zero docs (161.0ms) > >> >> * test _all_docs POST error when multi-get is not a {'key': [...]} > >> >> structure (104.3ms) > >> >> * test Regression test for COUCHDB-954 (skipped) > >> >> * test DELETE'ing a non-existent doc should 404 (100.0ms) > >> >> * test Revs info status is good (127.3ms) > >> >> * test PUT on existing DB should return 412 instead of 500 (97.6ms) > >> >> * test Database should be in _all_dbs (117.7ms) > >> >> * test Check for invalid document members (122.4ms) > >> >> * test Can create several documents (213.0ms) > >> >> * test Make sure you can do a seq=true option (99.1ms) > >> >> * test PUT doc has a Location header (skipped) > >> >> * test Create a document and save it to the database (116.3ms) > >> >> * test Created database has appropriate db info name (99.7ms) > >> >> * test PUT error when body not an object (89.5ms) > >> >> * test Simple map functions (473.0ms) > >> >> * test POST doc response has a Location header (117.1ms) > >> >> > >> >> CouchTestCase > >> >> > >> >> > >> >> Finished in 3.3 seconds > >> >> 24 tests, 0 failures, 3 skipped > >> >> > >> >> Randomized with seed 936284 > >> >
