On Mar 19, 2010, at 7:25 PM, Jan Lehnardt wrote:
>
> On 19 Mar 2010, at 18:07, J Chris Anderson wrote:
>
>>
>> On Mar 19, 2010, at 11:43 AM, Paul Davis wrote:
>>
>>> On Fri, Mar 19, 2010 at 2:02 PM, Jan Lehnardt <[email protected]> wrote:
>>>>
>>>> On 19 Mar 2010, at 12:50, Noah Slater wrote:
>>>>
>>>>>
>>>>> On 19 Mar 2010, at 17:11, Jan Lehnardt wrote:
>>>>>
>>>>>> We want to test the CouchDB code, not the browser's HTTP handling.
>>>>>
>>>>> Sure, but as one of CouchDB's primary interfaces is the browser, it seems
>>>>> to makes sense that we would want to test how this works. Testing from
>>>>> the browser allows us to test for and catch problems introduced by
>>>>> caching, etc - which is what our real world users would be running into.
>>>>>
>>>>> Unless I'm missing something?
>>>>
>>>> I fully agree, but we should have a separate browser interaction
>>>> suite for that. The test suite is a very untypical browser client and
>>>> doesn't really test real-world browser use-cases.
>>>>
>>>> Cheers
>>>> Jan
>>>> --
>>>
>>> +a bajillion.
>>>
>>
>> I prefer the browser tests because I'm much happier with JavaScript.
>
> I'm not saying we should get rid of the browser tests. But intermittent errors
> in the current test suite are not to be worried about to block a release.
I agree with all the comments about separation of tests and so forth. This
particular changes test is not intermittent, it consistently fails (on my
machine), enough that it's a pleasant surprise when it succeeds. When running
from the CLI I get the following:
not ok 10 changes expected '3', got '1'
When running in FF I also get the message above and occasionally:
• Exception raised:
{"message":"JSON.parse","fileName":"http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0","lineNumber":154,"stack":";(false)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:154\u000arun(-2)@http://127.0.0.1:5984/_utils/script/couch_test_runner.js?0.11.0:83\u000a"}
I haven't looked into it closely to find the root cause, it might just be the
test, but it's definitely not intermittent. From the CLI it happens almost
always
>
> If we want proper browser client testing, we'd need an additional test suite
> that covers common and uncommon use-cases. I believe the current test
> suite is as untypical as a browser client can be.
>
> Cheers
> Jan
> --
>
>
>>
>> But maybe I'm crazy
>>
>>
>>> I think its important to maintain *some* tests in the browser to test
>>> its ability to use CouchDB as a client, but we should put more work
>>> into separating API tests and core tests.
>>>
>>> Also, Zed Shaw has a very informative (and colorful) description of
>>> confounding factors [1]. Its about two thirds of the way down under a
>>> heading of "Confounding, Confounding, Confounding."
>>>
>>> http://www.zedshaw.com/essays/programmer_stats.html
>>
>
