Regarding databases distributions, I just had a chat with Stefan Klinger, the guy who is working on a VM with all databases. He should be sending a progress status on the group soon.
On Wed, May 6, 2009 at 14:06, Jonathan Pryor <[email protected]> wrote: > First, for most databases more tests pass than fail, so use [Category] to > mark tests that should pass would result in *more* methods being > attributed than marking those that fail (which is why I suggested marking > failing tests). > > As for tests failing on some vendors but not others, we just need to use > multiple categories with separate strings: > > [Test] > [Category("NotSqlite")] > [Category("NotSqlServer")] > public void TestSomething() {...} > > - Jon > > > On Wed, 2009-05-06 at 08:51 +0200, Giacomo Tesio wrote: > > Just another thought: > We could also use [Category] to mark "ShouldPass" tests. > Then all regressions in the ShouldPass category would be visible. > > BTW, since tests are shared among vendors, some "ShouldPass" for a vendor > but not yet for another... > > How to handle this? > > > Giacomo > > > On Wed, May 6, 2009 at 8:44 AM, Giacomo Tesio <[email protected]> wrote: > > [3] write a "simple" software monitoring checkins and running all tests at > each checkin. When it notes some regression, it would mail all the > developers a notification. > > BTW I've to note that > 1) Some test are wrong coded: they do not really test what they should > (noticed on some of the 101 modified tests) passing even if they should not > (some time using workaround against missing DbLinq feature... but than why > unit testing?). > 2) Some test don't passes just becouse the underlying db contains different > data set (and data structure). > > > Even if we could not fix all the DbLinq's errors we SHOULD fix the unit > tests ones. > Those error are the real noise we should fix. > > Moreover we really need a distributable testing infrastructure, urgently. > I simply can not run all tests, since I've not all the databases. > > > Giacomo > > > > On Wed, May 6, 2009 at 6:36 AM, Jonathan Pryor <[email protected]> wrote: > > I think we have a fundamental problem with our unit tests: most of them > don't have 100% pass rates. This is a problem because you can introduce > errors without realizing it, because the NUnit tree doesn't look > significantly different (lots of green and red before, vs. lots of green and > red later). > > For example, when I write the unit test page ( > http://groups.google.com/group/dblinq/web/unit-tests)<http://groups.google.com/group/dblinq/web/unit-tests%29>I > wrote that SQL Server had 425 tests run with 70 failures. As I write this > 432 tests are run (yay) with *190* failures (boo!) -- more than double the > error count compared to March 23. > > I don't know what caused the increased failures. Currently, I don't care. > > What I do currently care about is preventing such regressions in the > future, and the way to do that is by getting our unit tests 100% green (so > that regressions and errors are actually visible, not hidden in a sea of > existing errors). > > I can think of two[0] solutions to this, and I'm welcome to any additional > suggestions. > > 1. Use #if's in the test code to remove failing tests. > > 2. Use [Category] attributes on the tests to declare which tests shouldn't > be executed. The Categories tab within NUnit allows you to specify which > categories are executed. > > The problem with (1) is a gigantic increase in code complexity, as each > vendor will have a different set of failing unit tests, so we'd potentially > need checks for every vendor on each method. This is incredibly bad. > > (2) would at least avoid the "line noise" implied by #if, though it could > also get very "busy" (with upwards of 7 [Category] attributes on a given > method). > > Thoughts? Alternatives? > > Thanks, > - Jon > > [0] OK, a third solution would be to actually fix all the errors so that > everything is green without using (1) or (2) above, but I don't think that > this is practical in the short term. > > > > > > > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "DbLinq" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/dblinq?hl=en -~----------~----~----~----~------~----~------~--~---
