On 21 Mar 2010, at 12:24, Robert Dionne wrote: > > > > > On Mar 21, 2010, at 1:16 PM, Jan Lehnardt wrote: > >> >> On 21 Mar 2010, at 12:10, Noah Slater wrote: >> >>> What are the CLI tests, if not the etap tests? Are they integrated into the >>> build system? >> >> The CLI tests are the same as the browser tests, just run through our >> couchjs binary >> that has custom HTTP extensions to make the xhr work. At this point I don't >> think it >> is reliable enough to mimic browser behaviour and that we shouldn't use it >> for vetting >> the state of the code. > > This is likely true, but in this particular case I think there's a bug in the > changes code (that I'm trying to dig out). It's nice that it works on your > machine but on my machine, using FF, it fails often enough. Moreover it's > been around for a long time now so I figure it's worth figuring out. > > I don't have a dog in this fight (.ie. a paying customer) so this failure > doesn't bother me. With respect to policy, given the various bogocities of > browsers, I'd recommend something like these CLI tests plus the etaps ought > to be the "official" tests for vetting, and part of the build
Not that I disagree, but part (most?) of the appeal of the browser based tests are that they run in a real-world client instead of the lab that is couchjs+http :) Cheers Jan -- > > >> >> It is very useful when developing new code to not have to switch to and >> reload the >> browser over and over again. >> >> Cheers >> Jan >> -- >> >> >> >> >>> >>> On 21 Mar 2010, at 17:05, Jan Lehnardt wrote: >>> >>>> >>>> On 21 Mar 2010, at 06:04, Robert Dionne wrote: >>>> >>>>> On Mar 21, 2010, at 4:00 AM, Jan Lehnardt wrote: >>>>> >>>>>> >>>>>> On 20 Mar 2010, at 20:06, Paul Davis wrote: >>>>>> >>>>>>> On Sat, Mar 20, 2010 at 2:31 PM, Noah Slater <[email protected]> wrote: >>>>>>>> I think faulty test case should block the release, if I am to have any >>>>>>>> future sanity preparing releases. I don't want to delay and longer, so >>>>>>>> if >>>>>>>> you guys are absolutely sure this is a test error and not code error, >>>>>>>> then I >>>>>>>> propose that the test be commented out. Our tests form a contract >>>>>>>> between >>>>>>>> us, internally, and our users. If that contract has a bug, it should be >>>>>>>> removed or fixed - or it simply dilutes the importance of contract. If >>>>>>>> some >>>>>>>> one comments out the test, and we agree it is not indicative of an >>>>>>>> important >>>>>>>> bug, I can call the vote within hours. >>>>>>>> >>>>>>> >>>>>>> I'd have to agree on this. From the point of view of a release, if a >>>>>>> test reports a failure then it should be made to not report a failure. >>>>>>> If that's accomplished by disabling it, then there will be a commit >>>>>>> with a message that explains why it was disabled and etc and such on >>>>>>> and so forth. >>>>>> >>>>>> I'd do that if the test was failing for me :) >>>>> >>>>> it's not failing for you when you run changes.js with the CLI ? Fails >>>>> for me every time. >>>> >>>> I don't consider the CLI tests as part of the official test suite just yet. >>>> >>>> Cheers >>>> Jan >>>> -- >>>> >>>>> >>>>> Anyway I poked at this a bit yesterday and am not 100% sure the issue is >>>>> in the test. I tried putting a sleep in with no luck. If my understanding >>>>> of the JS is correct, CouchDB is supposed to be synchronous so it's not >>>>> timing. >>>>> >>>>> If someone could comment on the test itself it would be helpful. The >>>>> section of the code that fails: >>>>> >>>>> // changes get all_docs style with deleted docs >>>>> var doc = {a:1}; >>>>> db.save(doc); >>>>> db.deleteDoc(doc); >>>>> var req = CouchDB.request("GET", >>>>> "/test_suite_db/_changes?filter=changes_filter/bop&style=all_docs"); >>>>> var resp = JSON.parse(req.responseText); >>>>> TEquals(3, resp.results.length, "should return matching rows"); >>>>> >>>>> >>>>> seems odd to me. all_docs as I read the code will return docs with >>>>> deletes and conflicts but in this call the filter bop will not apply to >>>>> the doc {a:1} so I'm not sure what this delete prior to the call is >>>>> about. Anyway I can make it fail in the debugger so perhaps I can find >>>>> the root cause. >>>>> >>>>> >>>>> >>>>> >>>>>> >>>>>> Cheers >>>>>> Jan >>>>>> -- >>>>>> >>>>> >>>> >>> >> >
