Ah - my mistake, sorry: I didn't merge it after all. The PR is outstanding at https://github.com/apache/couchdb-jiffy/pull/3, with some discussion between @kxepal and myself. There's a note to Paul there asking wiser heads than mine to make a decision on whether to add it upstream.
Nick On Wed, 13 Jul 2016 at 18:00 Nick North <[email protected]> wrote: > Maybe I did something stupid with the PR. If I can get to a PC this > evening I'll take a look. > > Nick > On Wed, 13 Jul 2016 at 17:41, Paul Davis <[email protected]> > wrote: > >> That Jiffy error does seem like a conversion issue either from integer >> widths or endianess. I don't remember seeing a patch for that on >> Windows and looking back I'm not seeing anything from after 2013 that >> looks related (in davisp/jiffy). Looking at the apache/jiffy repo we >> use merges so I'd think it'd be unlikely to have been overwritten but >> I guess its always possible. >> >> Paul >> >> On Wed, Jul 13, 2016 at 3:31 AM, Nick North <[email protected]> wrote: >> > I think I submitted a PR for that Jiffy problem a while back: it's some >> > 32/64 bit issue. But I thought I'd also merged it. Unfortunately I'm >> away >> > from any useful tools for the next few days so can't check myself, but >> take >> > a look at the activity on the couchdb-jiffy repository. It's possible >> that >> > there has been a pull from upstream that overwrote my change. >> > >> > Nick >> > >> > On Wed, 13 Jul 2016 at 09:24 Joan Touzet <[email protected]> wrote: >> > >> >> I took a look at the eunit failures and found that the entire >> >> couchdb_os_daemon_test module fails due to issues with how it expects >> >> to launch daemons. >> >> >> >> The main issue is that the primary test harness is a .escript file, >> which >> >> on *nix is magically parsed via the #!/usr/bin/escript header. On >> Windows >> >> we are just trying to directly launch the .escript file which fails >> (since >> >> Windows has no idea how to execute *.escript files). There are also .sh >> >> scripts that are part of the test harness that will not run correctly. >> >> >> >> I've submitted a PR to simply bypass this entire test module for now. >> It >> >> sure would be swell to make it work but it'll be a fair bit of >> fiddling, >> >> especially in a way that makes eunit happy, to get it to work. >> >> >> >> Besides, there are other tests that drive the mrview os daemon; we will >> >> see massive failures in the JS test suite if the entire os daemon >> launching >> >> process fails. >> >> >> >> https://github.com/apache/couchdb-couch/pull/184 hopefully will land >> soon >> >> allowing the main couch eunit tests to pass. >> >> >> >> More worryingly we have a failure in jiffy on Windows: >> >> >> >> >> >> >> https://gist.github.com/anonymous/c796d4b048efa90b17b1f43008c59783#file-gistfile1-txt-L381-L389 >> >> >> >> Anyone who can help look at this one? (Paul?) >> >> >> >> -Joan >> >> >> >> ----- Original Message ----- >> >> From: "Joan Touzet" <[email protected]> >> >> To: [email protected] >> >> Sent: Tuesday, July 12, 2016 6:02:12 PM >> >> Subject: Re: 2.0 & Windows: status update >> >> >> >> Thanks, Paul. I'm starting to look at this today. >> >> >> >> In better news, current Windows JS tests now match the *nix JS test >> >> results. >> >> Only one test, replication.js, is failing (ignoring the ignored tests). >> >> >> >> Results: >> >> https://gist.github.com/anonymous/8e236848a89af440d3c56569e81f4829 >> >> >> >> Mr. Newson is looking at this failure right now and says we may be able >> >> to improve upon the testing methodology. >> >> >> >> -Joan >> >> >> >> ----- Original Message ----- >> >> From: "Paul Davis" <[email protected]> >> >> To: [email protected] >> >> Sent: Tuesday, July 12, 2016 4:37:15 PM >> >> Subject: Re: 2.0 & Windows: status update >> >> >> >> The logs posted at [1] show that we're seeing OS processes die with an >> >> exit code of 4. The most likely place I can find that that comes from >> >> is couchspawnkillable_win.c [2] which is nicely Windows specific so >> >> would do a lot to explain why we don't see it on *nix systems. >> >> Unfortunately other than pointing out that the subprocess creation >> >> seems to fail I don't have any idea or suggestion on how to debug >> >> further. >> >> >> >> [1] https://gist.github.com/anonymous/f2a94234195f007c3049e27d942482c1 >> >> [2] >> >> >> https://github.com/apache/couchdb-couch/blob/master/priv/spawnkillable/couchspawnkillable_win.c#L106 >> >> >> >> On Wed, Jun 29, 2016 at 4:21 PM, Sebastian Rothbucher >> >> <[email protected]> wrote: >> >> > Hi all, >> >> > >> >> > just to follow up on that: there is another PR coming up ( >> >> > https://github.com/apache/couchdb/pull/427) that tests for some more >> >> fixes >> >> > and brings even more stability. In the meantime, deleting dev/lib is >> >> indeed >> >> > the best way to produce reliable results. So is switching between >> >> > auth-tests-wip and master. But there's progress => it might all end >> up >> >> > nicely on master. >> >> > >> >> > Good luck, thanks & best >> >> > Sebastian >> >> > >> >> > >> >> > On Mon, Jun 20, 2016 at 5:26 PM, Nick North <[email protected]> >> wrote: >> >> > >> >> >> Thanks Sebastian. I'm looking at eunit at the moment, but hope to >> come >> >> back >> >> >> to these. >> >> >> >> >> >> Nick >> >> >> >> >> >> On Sun, 19 Jun 2016 at 23:01 Sebastian Rothbucher < >> >> >> [email protected]> wrote: >> >> >> >> >> >> > Hi Joan, Nick, >> >> >> > >> >> >> > the following gist provides a current run of the test against the >> >> latest >> >> >> > master of CouchDB - and the latest tests (from the auth-tests-wip >> >> >> branch): >> >> >> > >> >> >> > >> >> >> >> >> >> https://gist.github.com/sebastianrothbucher/efa3a992bd4de9996b4125da82a7e0de >> >> >> > Maybe you can use them >> >> >> > >> >> >> > Here's what I did to get both latest tests and latest code: >> >> >> > git checkout master >> >> >> > ./configure -c --disable-docs --disable-fauxton >> >> >> > make clean >> >> >> > make >> >> >> > git checkout auth-tests-wip >> >> >> > >> >> >> > Currently, make javascript seems not optimal as one tests (needs >> >> >> > investigation) seems to mess up the setup for all that's >> following. >> >> >> Hence, >> >> >> > I took this drastic measure to produce the logs: >> >> >> > >> >> >> > for t in test/javascript/tests/*.js; do rm -rf dev/lib; dev/run >> -n 1 >> >> -q >> >> >> > --with-admin-party-please test/javascript/run $t 2>&1 | tee -a >> >> >> jstest2.log; >> >> >> > done >> >> >> > >> >> >> > Maybe it makes sense for you to start w/ something similar to >> produce >> >> >> some >> >> >> > meaningful results. >> >> >> > >> >> >> > Best >> >> >> > Sebastian >> >> >> > >> >> >> > On Sun, Jun 19, 2016 at 6:04 PM, Nick North <[email protected]> >> >> wrote: >> >> >> > >> >> >> > > Sorry - I meant a single node cluster in that last message. And >> I >> >> meant >> >> >> > to >> >> >> > > sign my name correctly. >> >> >> > > >> >> >> > > Nick >> >> >> > > >> >> >> > > On Sun, 19 Jun 2016 at 16:56 Nick North <[email protected]> >> wrote: >> >> >> > > >> >> >> > > > I'm trying these tests now, and find that there are still a >> lot >> >> of JS >> >> >> > > > failures with a single cluster. Many of them look suspiciously >> >> >> similar >> >> >> > at >> >> >> > > > an initial glance, but I hope to look in more detail tomorrow. >> >> >> > > > >> >> >> > > > Nicj >> >> >> > > > >> >> >> > > > On Sun, 19 Jun 2016 at 15:20 Jan Lehnardt <[email protected]> >> wrote: >> >> >> > > > >> >> >> > > >> >> >> >> > > >> > On 17 Jun 2016, at 22:48, Joan Touzet <[email protected]> >> >> wrote: >> >> >> > > >> > >> >> >> > > >> > Hello everyone, >> >> >> > > >> > >> >> >> > > >> > I'd like to update the community on the status of the 2.0 >> port >> >> to >> >> >> > > >> Microsoft Windows. There are three parts to this email: the >> build >> >> >> > > >> tools/chain themselves, support in CouchDB for the Windows >> build >> >> >> > > process, >> >> >> > > >> and testing results. I'll cover them in that order. >> >> >> > > >> > >> >> >> > > >> > -Joan >> >> >> > > >> > >> >> >> > > >> > Build Tools/Chain >> >> >> > > >> > ================= >> >> >> > > >> > ** TL;DR: New glazier repo to join couchdb, contains >> scripts >> >> and >> >> >> > > README >> >> >> > > >> to build CouchDB 2.0 on Windows. >> >> >> > > >> > >> >> >> > > >> > Our work to date has been going on in Dave Cottlehuber's >> >> glazier >> >> >> > > >> repository at >> >> >> > > >> > >> >> >> > > >> > https://github.com/dch/glazier/tree/release/couchdb_2.0 >> >> >> > > >> > >> >> >> > > >> > The reason for the extra repository is that the Windows >> build >> >> >> > process >> >> >> > > >> is *very* ugly, involving 3 distinct build chains (Visual >> Studio, >> >> >> > Cygwin >> >> >> > > >> and the Mozilla Build system) to build all of the necessary >> >> >> > > prerequisites. >> >> >> > > >> The repository includes a number of support scripts to set up >> >> that >> >> >> > > >> environment, a README with a detailed walkthrough, and some >> >> patches >> >> >> > > >> necessary to the prerequisites to get them to build under the >> >> modern >> >> >> > > >> Windows b uild system. >> >> >> > > >> > >> >> >> > > >> > Parenthetically, it _is_ possible to use binary installs >> for >> >> the >> >> >> > > >> prerequisites (OpenSSL, libcurl, Erlang, SM 1.8.5), but Dave, >> >> Nick >> >> >> > North >> >> >> > > >> and I have evolved the glazier system over a number of years >> and >> >> >> it's >> >> >> > > >> proven quite effective. Plus, we don't have to worry about >> the >> >> >> > > provenance >> >> >> > > >> of any of the binaries since we build everything from source >> >> >> directly, >> >> >> > > and >> >> >> > > >> that's important when we put up an unofficial Windows build >> for >> >> >> > > download at >> >> >> > > >> https://couchdb.apache.org/ . >> >> >> > > >> > >> >> >> > > >> > Good news: as of today I've requested and Infra has >> created a >> >> new >> >> >> > > >> apache couchdb-glazier repo, and it's my intent to mirror >> >> >> dch/glazier >> >> >> > > over >> >> >> > > >> into the ASF's repo once things have stabilized a bit more >> (PR >> >> and >> >> >> > > merge of >> >> >> > > >> the release/couchdb_2.0 branch, and pending progress on >> steps 2 >> >> and >> >> >> 3 >> >> >> > > >> below). Dave and I did an audit of the repository as it >> stands, >> >> and >> >> >> > > since >> >> >> > > >> all checkins come from CouchDB contributors already, we are >> good >> >> to >> >> >> go >> >> >> > > from >> >> >> > > >> a licensing perspective. >> >> >> > > >> > >> >> >> > > >> > >> >> >> > > >> > Overall CouchDB Windows support >> >> >> > > >> > =============================== >> >> >> > > >> > ** TL;DR: Windows support in 2.0 a priority, conversion of >> >> >> top-level >> >> >> > > >> Makefile in progress. >> >> >> > > >> > >> >> >> > > >> > There are two aspects to native CouchDB Windows support. >> The >> >> first >> >> >> > is >> >> >> > > >> anything within the CouchDB code itself that assumes a >> Unix-like >> >> >> > > >> environment. Fortunately, most of these problems have been >> worked >> >> >> out >> >> >> > in >> >> >> > > >> prior releases. I'm not aware of any outstanding issues here >> >> (except >> >> >> > one >> >> >> > > >> point below under test results). >> >> >> > > >> > >> >> >> > > >> > The other aspect is the build setup within the couchdb repo >> >> >> itself. >> >> >> > > >> I've already converted the bash configure script into a >> >> PowerShell >> >> >> > > >> configure script that works fine. However, the Makefile has >> >> bashisms >> >> >> > in >> >> >> > > it >> >> >> > > >> and assumes GNU Make. I've started a conversion of this into >> >> Windows >> >> >> > > NMake >> >> >> > > >> format and will submit a PR for a Makefile.win in due course. >> >> >> > > >> > >> >> >> > > >> > I want to answer two frequent questions we get here before >> they >> >> >> get >> >> >> > > >> re-asked: >> >> >> > > >> > >> >> >> > > >> > 1) Why not use a cygwin environment to retain >> compatibility >> >> with >> >> >> > the >> >> >> > > >> Unix build process? The answer is that performance suffers, >> the >> >> >> build >> >> >> > > chain >> >> >> > > >> is onerous, there are link-time problems when trying to link >> >> against >> >> >> > > things >> >> >> > > >> built using Visual Studio, and there are still assumptions on >> >> paths >> >> >> > that >> >> >> > > >> don't work out. We can't get away from making >> Windows-specific >> >> >> > > >> customizations to the build process anyway, so we might as >> well >> >> take >> >> >> > the >> >> >> > > >> extra step and support the build process properly. It's not >> THAT >> >> >> much >> >> >> > > work >> >> >> > > >> to convert the Makefile and configure script, and our >> top-level >> >> >> > Makefile >> >> >> > > >> really isn't much more than a shell script anyway (every >> target >> >> is a >> >> >> > > .PHONY >> >> >> > > >> target!). In fact, a TODO for an enterprising developer >> might be >> >> to >> >> >> > > rewrite >> >> >> > > >> our top-level Makefile/Makefile.win as a Python script that >> "does >> >> >> the >> >> >> > > right >> >> >> > > >> thing" on both platforms, the same way our dev/run script >> works >> >> >> today. >> >> >> > > >> > >> >> >> > > >> > 2) Why not use the new "Bash and Ubuntu on Windows" >> >> functionality >> >> >> > > >> Microsoft has announced for Windows 10? There are two >> distinct >> >> >> > problems >> >> >> > > >> here. The first is that there is a very large install base >> still >> >> of >> >> >> > > Windows >> >> >> > > >> 7 and 8 (and Windows Server) machines that cannot run this >> >> >> subsystem. >> >> >> > > The >> >> >> > > >> second is that Microsoft themselves say this about the >> >> >> functionality: >> >> >> > > >> > >> >> >> > > >> > "Second, while you’ll be able to run native Bash and >> many >> >> >> Linux >> >> >> > > >> command-line tools on Windows, it’s important to note that >> this >> >> is a >> >> >> > > >> developer toolset to help you write and build all your code >> for >> >> all >> >> >> > your >> >> >> > > >> scenarios and platforms. This is not a server platform upon >> which >> >> >> you >> >> >> > > will >> >> >> > > >> host websites, run server infrastructure, etc." >> >> >> > > >> > >> >> >> > > >> > Given this strong warning from Microsoft themselves (which >> >> hints >> >> >> at >> >> >> > > >> performance consideratings), and the fact that download >> >> statistics >> >> >> > show >> >> >> > > an >> >> >> > > >> equal number of downloads of the CouchDB .tar source and the >> >> Windows >> >> >> > > .zip >> >> >> > > >> installer from our couchdb.apache.org website, we need to >> >> consider >> >> >> > that >> >> >> > > >> people are running CouchDB on Windows not just as a developer >> >> tool >> >> >> but >> >> >> > > as a >> >> >> > > >> fully-fledged server. As such it behooves us to build it >> >> "properly" >> >> >> > as a >> >> >> > > >> normal Windows binary/service. >> >> >> > > >> > >> >> >> > > >> >> >> >> > > >> Great progress Joan! Thank you! :) >> >> >> > > >> >> >> >> > > >> > Test Results >> >> >> > > >> > ============ >> >> >> > > >> > ** TL;DR: Lots of things are failing. Joan needs help >> >> interpreting >> >> >> > the >> >> >> > > >> results or she will go around the bend. >> >> >> > > >> > >> >> >> > > >> > Here are the current test results in gist form. >> >> >> > > >> > >> >> >> > > >> > EUnit: >> >> >> > > >> >> >> https://gist.github.com/anonymous/3203ed27c60cf3da4f0f0d5bff731722 >> >> >> > > >> > >> >> >> > > >> > JS tests: >> >> >> > > >> >> >> https://gist.github.com/anonymous/93b0b70ed445ca4043a63140f8d219bf >> >> >> > > >> > >> >> >> > > >> > For the EUnit tests, everything other than os_process stuff >> >> seems >> >> >> to >> >> >> > > be >> >> >> > > >> working. Honestly, I think we can release without os_process >> >> support >> >> >> > on >> >> >> > > >> Windows, though I should file a bug to track this. I am >> actually >> >> >> > > inclined >> >> >> > > >> to disable os_process support on Windows and the related >> eunit >> >> tests >> >> >> > > rather >> >> >> > > >> than fix this rarely-needed functionality, unless someone on >> this >> >> >> list >> >> >> > > >> objects. >> >> >> > > >> >> >> >> > > >> You are probably thinking about CouchDB Externals, which >> >> definitely >> >> >> > use >> >> >> > > >> os_process functionality and which I’d also be fine with >> dropping >> >> >> > > support >> >> >> > > >> for Windows for now, but os_process is also used by the view >> >> server, >> >> >> > so >> >> >> > > we >> >> >> > > >> should definitely get them passing. >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> > For the JS tests, a *lot* is failing. I need to know how >> much >> >> this >> >> >> > > >> differs from a Linux/OSX baseline today, can anyone help me >> >> follow >> >> >> up >> >> >> > > here? >> >> >> > > >> >> >> >> > > >> Can you try running these against a -n 1 cluster? We are not >> set >> >> up >> >> >> to >> >> >> > > >> run JS tests against more nodes at this point. >> >> >> > > >> >> >> >> > > >> On master/unix most if not all JS tests should either pass or >> >> >> skipped >> >> >> > > >> with a TODO message. >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> Best >> >> >> > > >> Jan >> >> >> > > >> -- >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> > >> >> >> >> >> >> >
