On 20 Mar 2010, at 10:26, Noah Slater wrote: > Jan, should this block the release? From what I can tell, it should.
I don't think a faulty test case should block the release. Cheers Jan -- > > > > On 20 Mar 2010, at 11:32, Robert Dionne <[email protected]> wrote: > >> >> >> >> On Mar 19, 2010, at 7:25 PM, Jan Lehnardt wrote: >> >>> >>> On 19 Mar 2010, at 18:07, J Chris Anderson wrote: >>> >>>> >>>> On Mar 19, 2010, at 11:43 AM, Paul Davis wrote: >>>> >>>>> On Fri, Mar 19, 2010 at 2:02 PM, Jan Lehnardt <[email protected]> wrote: >>>>>> >>>>>> On 19 Mar 2010, at 12:50, Noah Slater wrote: >>>>>> >>>>>>> >>>>>>> On 19 Mar 2010, at 17:11, Jan Lehnardt wrote: >>>>>>> >>>>>>>> We want to test the CouchDB code, not the browser's HTTP handling. >>>>>>> >>>>>>> Sure, but as one of CouchDB's primary interfaces is the browser, it >>>>>>> seems to makes sense that we would want to test how this works. Testing >>>>>>> from the browser allows us to test for and catch problems introduced by >>>>>>> caching, etc - which is what our real world users would be running into. >>>>>>> >>>>>>> Unless I'm missing something? >>>>>> >>>>>> I fully agree, but we should have a separate browser interaction >>>>>> suite for that. The test suite is a very untypical browser client and >>>>>> doesn't really test real-world browser use-cases. >>>>>> >>>>>> Cheers >>>>>> Jan >>>>>> -- >>>>> >>>>> +a bajillion. >>>>> >>>> >>>> I prefer the browser tests because I'm much happier with JavaScript. >>> >>> I'm not saying we should get rid of the browser tests. But intermittent >>> errors >>> in the current test suite are not to be worried about to block a release. >> >> I agree with all the comments about separation of tests and so forth. This >> particular changes test is not intermittent, it consistently fails (on my >> machine), enough that it's a pleasant surprise when it succeeds. When >> running from the CLI I get the following: >> >> not ok 10 changes expected '3', got '1' >> >> When running in FF I also get the message above and occasionally: >> >> • Exception raised: >> {"message":"JSON.parse","fileName":"http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0","lineNumber":154,"stack":";(false)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:154\u000arun(-2)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:83\u000a"} >> >> I haven't looked into it closely to find the root cause, it might just be >> the test, but it's definitely not intermittent. From the CLI it happens >> almost always >> >> >> >>> >>> If we want proper browser client testing, we'd need an additional test suite >>> that covers common and uncommon use-cases. I believe the current test >>> suite is as untypical as a browser client can be. >>> >>> Cheers >>> Jan >>> -- >>> >>> >>>> >>>> But maybe I'm crazy >>>> >>>> >>>>> I think its important to maintain *some* tests in the browser to test >>>>> its ability to use CouchDB as a client, but we should put more work >>>>> into separating API tests and core tests. >>>>> >>>>> Also, Zed Shaw has a very informative (and colorful) description of >>>>> confounding factors [1]. Its about two thirds of the way down under a >>>>> heading of "Confounding, Confounding, Confounding." >>>>> >>>>> http://www.zedshaw.com/essays/programmer_stats.html >>>> >>> >>
