First, for most databases more tests pass than fail, so use [Category]
to mark tests that should pass would result in more methods being
attributed than marking those that fail (which is why I suggested
marking failing tests).
As for tests failing on some vendors but not others, we just need to use
multiple categories with separate strings:
[Test]
[Category("NotSqlite")]
[Category("NotSqlServer")]
public void TestSomething() {...}
- Jon
On Wed, 2009-05-06 at 08:51 +0200, Giacomo Tesio wrote:
> Just another thought:
> We could also use [Category] to mark "ShouldPass" tests.
> Then all regressions in the ShouldPass category would be visible.
>
> BTW, since tests are shared among vendors, some "ShouldPass" for a
> vendor but not yet for another...
>
> How to handle this?
>
>
> Giacomo
>
>
>
> On Wed, May 6, 2009 at 8:44 AM, Giacomo Tesio <[email protected]>
> wrote:
>
> [3] write a "simple" software monitoring checkins and running
> all tests at each checkin. When it notes some regression, it
> would mail all the developers a notification.
>
> BTW I've to note that
> 1) Some test are wrong coded: they do not really test what
> they should (noticed on some of the 101 modified tests)
> passing even if they should not (some time using workaround
> against missing DbLinq feature... but than why unit testing?).
> 2) Some test don't passes just becouse the underlying db
> contains different data set (and data structure).
>
>
> Even if we could not fix all the DbLinq's errors we SHOULD fix
> the unit tests ones.
> Those error are the real noise we should fix.
>
> Moreover we really need a distributable testing
> infrastructure, urgently.
> I simply can not run all tests, since I've not all the
> databases.
>
>
> Giacomo
>
>
>
>
>
> On Wed, May 6, 2009 at 6:36 AM, Jonathan Pryor
> <[email protected]> wrote:
>
> I think we have a fundamental problem with our unit
> tests: most of them don't have 100% pass rates. This
> is a problem because you can introduce errors without
> realizing it, because the NUnit tree doesn't look
> significantly different (lots of green and red before,
> vs. lots of green and red later).
>
> For example, when I write the unit test page
> (http://groups.google.com/group/dblinq/web/unit-tests)
> I wrote that SQL Server had 425 tests run with 70
> failures. As I write this 432 tests are run (yay)
> with 190 failures (boo!) -- more than double the error
> count compared to March 23.
>
> I don't know what caused the increased failures.
> Currently, I don't care.
>
> What I do currently care about is preventing such
> regressions in the future, and the way to do that is
> by getting our unit tests 100% green (so that
> regressions and errors are actually visible, not
> hidden in a sea of existing errors).
>
> I can think of two[0] solutions to this, and I'm
> welcome to any additional suggestions.
>
> 1. Use #if's in the test code to remove failing tests.
>
> 2. Use [Category] attributes on the tests to declare
> which tests shouldn't be executed. The Categories tab
> within NUnit allows you to specify which categories
> are executed.
>
> The problem with (1) is a gigantic increase in code
> complexity, as each vendor will have a different set
> of failing unit tests, so we'd potentially need checks
> for every vendor on each method. This is incredibly
> bad.
>
> (2) would at least avoid the "line noise" implied by
> #if, though it could also get very "busy" (with
> upwards of 7 [Category] attributes on a given method).
>
> Thoughts? Alternatives?
>
> Thanks,
> - Jon
>
> [0] OK, a third solution would be to actually fix all
> the errors so that everything is green without using
> (1) or (2) above, but I don't think that this is
> practical in the short term.
>
>
>
>
>
>
>
>
>
> >
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"DbLinq" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/dblinq?hl=en
-~----------~----~----~----~------~----~------~--~---
