+1 on all the stuff Paul said. On Thu, Oct 20, 2011 at 9:25 PM, Robert Newson <[email protected]> wrote:
> I'll also note that the bug that killed round 1 of 1.1.1 was not found > by any test we currently have. All it would have taken is a test that > did any map call followed by almost any other bit of javascript (and > sm 1.7.0). > > On 20 October 2011 21:22, Paul Davis <[email protected]> wrote: > > On Thu, Oct 20, 2011 at 1:45 PM, Randall Leeds <[email protected]> > wrote: > >> On Thu, Oct 20, 2011 at 13:42, Paul Davis <[email protected] > >wrote: > >> > >>> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds <[email protected]> > wrote: > >>> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane < > >>> [email protected]>wrote: > >>> > > >>> >> > >>> >> > For what it's worth, a CLI based test system is what I was > imagining > >>> >> > as well. Take Futon out of the mix and test CouchDB. > >>> >> > >>> >> IMO, If CouchDB is intended to be a server that can be accessed from > >>> >> the browser directly, then there should continue to be some kind of > >>> >> browser-based test suite that would serve to confirm this > capability. > >>> >> > >>> >> > >>> >> I have been looking closely at the Futon tests in 1.1.0 for the last > >>> >> several days, with the idea that I might begin to clean them up a > bit > >>> >> as time permits. > >>> >> > >>> >> I have found that, while some of these test failures are totally > bogus, > >>> >> *some* of them actually do stem from real issues -- minor > >>> >> incompatibilities between CouchDB's http interface, and the internal > >>> >> mechanisms of modern browsers (XHR, caching, etc). > >>> >> > >>> >> These are problems that we're not going to catch with a stateless, > >>> >> cache-less http client running on the CLI. (I can provide examples) > >>> >> > >>> >> These issues have the potential to cause real problems for > >>> >> developers of real browser-based apps "in the wild". That means, > >>> >> there's valuable info to be gathered from the browser tests, Iff we > >>> >> can clean them up, and make them behave consistently; so that > >>> >> when they fail or succeed, we can actually trust the results. > >>> >> > >>> >> > >>> >> After digging around a good bit, I can see no reason why the > existing > >>> >> tests couldn't be cleaned up and made to work correctly in all > current > >>> >> versions of major browsers. I also see no reason why the same tests > >>> >> couldn't be used successfully from the CLI and `make check` as well. > >>> >> > >>> >> I do see significant benefits to using the same javascript test code > in > >>> >> all environments we test. > >>> >> > >>> >> -Lee > >>> >> (irc: coltr) > >>> >> > >>> > > >>> > +1 > >>> > Verify Installation could grow into a suite of browser/futon tests > that > >>> > verify that futon (and apps in general) work, including interactions > with > >>> > proxies and the like. > >>> > >>> Sure. Client tests that test the client are fine. > >>> > >>> > The test suite for developers should run cleanly from the CLI as part > of > >>> > make check, but continue to be exposed in futon. We should work to be > >>> sure > >>> > they function as well as possible, for the reasons you provide. > >>> > > >>> > >>> Blargh no. Server tests should be testing the server. The entire point > >>> of moving to the command line is so that we don't have to maintain the > >>> Futon test suite. Just look at the 1.1.1 thread (or damn near any > >>> release thread) and the wildly varying reports of test output. The > >>> situation is just a waste of time for everyone involved. > >>> > >>> > I think the JS testing situation is a great place for people to jump > in > >>> and > >>> > help out, especially with the browser environment diversity. > >>> > > >>> > >>> Sure, but I don't see what this has to do with browsers. > >>> > >> > >> People who aren't into the internals can help to fix the suite to work > in > >> different browser environments. That's all I meant. > >> > > > > Seeing as I'm having a Negative Nancy day, I'll just ask rhetorically, > > "If these people exist, why do I not see anything in JIRA?" > > > >> I suggested that the CLI tests be exposed in Futon because I think there > are > >> probably some JS heads in this community who wouldn't have too much > trouble > >> fixing a lot of the user agent related issues in the test suite. I > didn't > >> mean to suggest that it should continue to be part of the release > procedure > >> (it shouldn't) or that we should feel 100% obligated to make sure they > pass > >> in 100% of environments (we can't and shouldn't), but J. Lee's point > about > >> how keeping such tests around can sometimes expose interesting problems > we > >> wouldn't otherwise see, possible outside the CouchDB codebase even, is > >> worthwhile. > >> > >> -Randall > >> > > > > We've had these tests for three years or more now. Perhaps I'm just > > being dense today but I can't think of a single specific case where > > testing things in the browser has lead to a bug report/fix that we > > wouldn't have found with pure CLI tests. > > > > The only thing that I'm aware that the tests have done for us is > > required us to exert a nontrivial amount of effort to keep them > > running in multiple browser environments. I have no interest in > > maintaing these as tests runnable in the browser. I want to create a > > CLI test environment that promotes stable, repeatable, concise tests. > > Running these in a browser is the antithesis to such an environment. > > >
