On Wed, Mar 24, 2010 at 17:03, Jonathan Pryor <[email protected]> wrote: > Perhaps the better solution is to remove tests/*.sql and document use of > examples/*/sql/*.sql. I'm not entirely sure. I'm leaning that way. Including the *.db3 binary blob.
> (There is something to be > said about smaller datasets, but at the same time some issues might only > be apparent in larger datasets, e.g. finding appropriate FK joins, etc. > I know that when fixing some of the tests last year that finding queries > that would "work"/test the desired behavior was somewhat more difficult > under the (then smaller) SQLite DB, as there was simply less data to > query...) You are right, that that there are more options with more data, rows could be added those missing cases. Issues like 195 and 196 are also hard to test with the current schema, but having few data, makes it easier to extend the schema. On the other hand, there are probably tables that aren't used by any tests. Using a large dataset in the hope of finding something, sounds more like integration testing or fuzz testing (which are of course also valid testing strategies) than unit testing. >> [...] the EntitySet.Refresh01 and .Refresh02 test takes >> about 2 minutes [...] using the full northwind database [...] > This sounds like a feature, not a bug -- it gives us an incentive to > profile wtf is going on there and fix it. :-) Also my first thought. But then your not really unit testing. >> > 2. You *MAY* want to consider an alternate table-generation approach. >> Aren't there a difference between "known-bad" in the sense that "this >> will never work for this vendor" (e.g. testing for store procedures in >> a database that doesn't support them), and failing tests that are >> failing just because the errors haven't been fixed yet. > > There should be a difference, but there is not currently such a > distinction made. This would be trivially supported by adding another > custom attribute (e.g. [NotSupported(PROVIDER)]), so that > [Explicit]+[NotSupported] means the DB can't support the test, while > just [Explicit] means that DbLinq is buggy. Ok, so the tests currently marked Explicit should just be read as regular failing tests and are todo. >> I don't assume >> the later type is supposed to be disabled. It would seem quite odd to >> me to disable tests just because they are failing. > > That is, in fact, the point -- to disable tests because they are > failing. The problem was this: there would be ~100 tests that would > always fail. I'd fix a bug, which should fix a test...and 110 tests > would now be failing. Which ones broke? Who knows! > > (It's actually knowable...through lots of grep+sed+etc. magic with > parsing the NUnit-generated TestResult.xml files, and remembering to > preserve the TestResult.xml file from before the fixes, and...) > > In short, it made progress *significantly* slower, because I had no way > of knowing which (previously working) tests my "fix" broke. > > Think of it as a developer time optimization. :-) Okay, I'm of course not in a position to tell you what works for you. But in moving the test report generation from the excel sheet to an app, it was also the idea that reports for tracking fixes and regressions over time both quantitatively and qualitatively could be added. I could try adding an option that given two TestResults.xml files would show the fixed and regressed tests. As for not "remembering to preserve the TestResult.xml file from before the fixes" isn't that kind of like failing to add/remove the Explicit option? >> BTW, I was wondering if it wouldn't be an idea to use categories for >> disabling tests instead of the "preprocessor directives" and >> "Explicit"-attribute > We actually had that discussion: > > http://groups.google.com/group/dblinq/browse_thread/thread/4242411770d568f4/10aab71e304ac24f?lnk=gst&q=category#10aab71e304ac24f > In particular: > http://groups.google.com/group/dblinq/msg/64eae3e2a410af11 > (In short: I tried. It didn't work. Perhaps I was an idiot and did it > wrong; you're welcome to try. ;-) Thanks for the pointers. No need for me to cover the same ground. There is also the problem that each vendor have its own Northwind-class which don't really have anything to do with the others (e.g. they don't implement a common interface). -- Anders -- You received this message because you are subscribed to the Google Groups "DbLinq" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/dblinq?hl=en.
