Committed. Now RELEASE unit tests track for regressions, while DEBUG ones make them run all.
Remember to unselect the DEBUG define on RELEASE compilation in each test project you touch. Giacomo PS: Someone should take a to EntitySetExpression usage: search "//if (!isEntitySet)" in ExpressionDispatcher.Analyzer.cs, uncomment it and make it work! I've started this work, but it's quite long and I've no time now... On Thu, Jun 4, 2009 at 9:24 AM, Giacomo Tesio <[email protected]> wrote: > Please take a look to the attached patch (against r1104). > > It enable ALL tests to be run by default when compiled in DEBUG, while keep > the behaviour you are introducing (the #if / [Explicit]) for the release. > > This would be logically correct (since we actually need to not get > regressions in release) AND confortable during development. > > We could keep just 1 NUnit open looking for regressions when the number of > failure increase AND before committing by simply selecting the NUnit-GUI -> > Project -> Configuration -> Release. > > > I find this really usefull since would help to identify regressions but > allow to faster identifing already fixed tests to correct #if statements. > > > Giacomo > > > > On Wed, Jun 3, 2009 at 12:17 AM, Giacomo Tesio <[email protected]> wrote: > >> I mean that falling test could be more useful than passing ones. >> >> Actually you do not want green tests, you want a way to track regressions >> easily. >> >> Green tests is one way, but has many issue (those you said plus assigning >> mantaining the attributes etc...) >> We should fix the test, but since its difficult in the short term, we are >> searching a work around. >> >> I think the work around would be not to not run failing tests, but to use >> nunit-console plus some script to look for regressions. >> I really think that such a script could be written in far less time of the >> #if / [Explicit] set up. >> >> >> Giacomo >> >> >> On Tue, Jun 2, 2009 at 8:13 PM, Jonathan Pryor <[email protected]> wrote: >> >>> OK, in an ideal world my SqlServerKnownFailureAttribute would work. >>> Alas, this isn't a perfect world. From NUnit's >>> NUnitCore/core/NUnitFramework.cs: >>> >>> public class NUnitFramework >>> { >>> public const string ExplicitAttribute = >>> "NUnit.Framework.ExplicitAttribute"; >>> // ... >>> public static void ApplyCommonAttribute(Attribute[] attributes, Test test) >>> { >>> //... >>> foreach (Attribute attribute in attributes) >>> { >>> // ... >>> string attributeName = attribute.GetType().FullName; >>> switch (attributeName) >>> // ... >>> case ExplicitAttribute: ... >>> } >>> } >>> } >>> >>> In short, instead of using Type.IsAssignableFrom() to see if a type ISA >>> ExplicitAttribute, it checks the type name directly, which (of course) >>> fails if you subclass the attribute. >>> >>> Thus my nifty subclass cleanup effort doesn't actually work. :-( >>> >>> This of course means that the only way to mark certain tests as >>> [Explicit] is to use #if within the tests themselves. (Or bundle a >>> forked copy of NUnit with DbLinq...which really isn't a great idea.) >>> >>> - Jon >>> >>> >>> On Tue, 2009-06-02 at 13:53 -0400, Jonathan Pryor wrote: >>> >>> I'm not sure what you mean by "lose control about failing [tests]." >>> [Explicit] tests can still be run from within the NUnit GUI, they just >>> need to be explicitly selected first. >>> >>> Though the "explicit selection" means that you can't easily run *all*tests >>> to see if any of the "known bad" tests are now working, which is >>> annoying. *However*, you can instead show the checkboxes (*V*iew -> *T*ree >>> -> Show Check*B*oxes), select the Explicit tests, and then Run will run >>> all tests including the explicitly checked ones. >>> >>> (For the really ideal case, there would be a way to select all checkboxes >>> at once, but this should work for now...) >>> >>> This is certainly better than using [Ignore], which provides *no*mechanism >>> to actually run the test without editing source code (that I could >>> find, anyway). >>> >>> Of course, this still has the downside of "uglying up" the test source >>> code with lots of #if code. An alternative would be to "centralize" via >>> attributes: >>> >>> [AttributeUsage(AttributeTargets.Method, AllowMultiple=true)] >>> class SqlServerKnownFailureAttribute : >>> #if MSSQL >>> NUnit.Framework.ExplicitAttribute >>> #else >>> Attribute >>> #endif >>> { >>> } >>> >>> // and similar for SqliteKnownFailure, etc. >>> >>> >>> This would allow tests such as: >>> >>> [SqlServerKnownFailure] >>> [Test] >>> public void FirstInternal02() {...} >>> >>> >>> (The #if within the SqlServerKnownFailureAttribute declaration would >>> prevent it from being an ExplicitAttribute within e.g. >>> DbLinq.Sqlite_test.dll, thus allowing the test to be run with other >>> vendors.) >>> >>> - Jon >>> >>> On Tue, 2009-06-02 at 08:44 +0100, Giacomo Tesio wrote: >>> >>> I'm not so sure that excluding tests would be a good idea, since we could >>> loose control about failing ones... >>> >>> Can you send me the PITA :-D >>> >>> I think that a script controlling for regressions would be a good >>> thing... >>> >>> >>> Giacomo >>> >>> On Tue, Jun 2, 2009 at 2:05 AM, Jonathan Pryor <[email protected]> wrote: >>> >>> After trying to actually do this, [Category("...")] is a non-starter. >>> >>> The requirements for what I want are simple: >>> >>> 1. Have a *single* .nunit file for all "interesting" tests. For >>> example, svn-trunk has DbLinq-All.nunit which contains *all* unit >>> tests, while DbLinq-Sqlite-Sqlserver.nunit contains only SQLite, >>> Microsoft SQL Server, and "no database" (*_ndb*) tests. The latter >>> exists because I'm currently only interested in SQLite and SQL Server >>> support (as I don't have any other databases available), so testing >>> anything >>> else is pointless. I want a *single* .nunit file so that I can hit >>> Run and run all tests in one window instead of needing to manage several >>> separate NUnit runner windows (and possibly forget to run some subset of >>> tests). >>> 2. *All* tests should be built, even the failing ones. This is so >>> that we can easily run previously failing tests to see if any are now >>> working. >>> 3. Known failing tests should *not* be run by default. >>> >>> (Why do I care about a green tree? See below [0].) >>> >>> So, solutions: >>> >>> 1. #if the tests, as originally mentioned. This fails requirement (2). >>> >>> 2. Use [Conditional("...")]. This fails (1), as *all* assemblies within >>> a .nunit file share the same set of categories, meaning that if I have >>> one set of tests that fail under SQLite, and a different (non-overlapping) >>> set of tests that fail under SQL Server, *both* sets won't be executed >>> if both categories are specified. This is obviously not good. >>> >>> 3. What I'm currently thinking is to use [Exclude] in combination with >>> #if, e.g.: >>> >>> #if SQLSERVER /* || other #defines */ >>> [Explicit] >>> #endif >>> public void TestName() {...} >>> >>> >>> This will fulfill my 3 requirements. Alternatively we could use >>> [Ignore] instead of [Explicit], though [Ignore] will make the test >>> runner turn yellow instead of green in an "all pass" scenario. >>> >>> - Jon >>> >>> [0] The problem I've been facing is a simple one: I'll see that there's >>> e.g. 190 failing tests. I'll fix one, and find that I now have 195 failing >>> tests (i.e. I fixed one test and regressed 6 others). >>> >>> Now, which tests regressed? :-) >>> >>> My current solution, which is a PITA, is a nasty combination of >>> grep+sed+diff on the "before svn changes" and "after svn changes" >>> TestResult.xml files, e.g.: >>> >>> grep '<test-case' TestRestult.previous.xml | sed 's/time=.*//g' > p >>> grep '<test-case' TestRestult.current.xml | sed 's/time=.*//g' > c >>> diff -u p c >>> >>> So I'm currently finding it of utmost importance to get a green build, >>> simply so I can more easily see which tests I'm regressing while trying to >>> fix things. >>> >>> >>> >>> On Wed, 2009-05-06 at 08:06 -0400, Jonathan Pryor wrote: >>> >>> First, for most databases more tests pass than fail, so use [Category] to >>> mark tests that should pass would result in *more* methods being >>> attributed than marking those that fail (which is why I suggested marking >>> failing tests). >>> >>> As for tests failing on some vendors but not others, we just need to use >>> multiple categories with separate strings: >>> >>> [Test] >>> [Category("NotSqlite")] >>> [Category("NotSqlServer")] >>> public void TestSomething() {...} >>> >>> - Jon >>> >>> On Wed, 2009-05-06 at 08:51 +0200, Giacomo Tesio wrote: >>> >>> Just another thought: >>> We could also use [Category] to mark "ShouldPass" tests. >>> Then all regressions in the ShouldPass category would be visible. >>> >>> BTW, since tests are shared among vendors, some "ShouldPass" for a vendor >>> but not yet for another... >>> >>> How to handle this? >>> >>> >>> Giacomo >>> >>> >>> On Wed, May 6, 2009 at 8:44 AM, Giacomo Tesio <[email protected]> wrote: >>> >>> [3] write a "simple" software monitoring checkins and running all tests >>> at each checkin. When it notes some regression, it would mail all the >>> developers a notification. >>> >>> BTW I've to note that >>> 1) Some test are wrong coded: they do not really test what they should >>> (noticed on some of the 101 modified tests) passing even if they should not >>> (some time using workaround against missing DbLinq feature... but than why >>> unit testing?). >>> 2) Some test don't passes just becouse the underlying db contains >>> different data set (and data structure). >>> >>> >>> Even if we could not fix all the DbLinq's errors we SHOULD fix the unit >>> tests ones. >>> Those error are the real noise we should fix. >>> >>> Moreover we really need a distributable testing infrastructure, urgently. >>> I simply can not run all tests, since I've not all the databases. >>> >>> >>> Giacomo >>> >>> >>> >>> On Wed, May 6, 2009 at 6:36 AM, Jonathan Pryor <[email protected]> wrote: >>> >>> I think we have a fundamental problem with our unit tests: most of them >>> don't have 100% pass rates. This is a problem because you can introduce >>> errors without realizing it, because the NUnit tree doesn't look >>> significantly different (lots of green and red before, vs. lots of green and >>> red later). >>> >>> For example, when I write the unit test page ( >>> http://groups.google.com/group/dblinq/web/unit-tests)<http://groups.google.com/group/dblinq/web/unit-tests%29>I >>> wrote that SQL Server had 425 tests run with 70 failures. As I write this >>> 432 tests are run (yay) with *190* failures (boo!) -- more than double >>> the error count compared to March 23. >>> >>> I don't know what caused the increased failures. Currently, I don't >>> care. >>> >>> What I do currently care about is preventing such regressions in the >>> future, and the way to do that is by getting our unit tests 100% green (so >>> that regressions and errors are actually visible, not hidden in a sea of >>> existing errors). >>> >>> I can think of two[0] solutions to this, and I'm welcome to any >>> additional suggestions. >>> >>> 1. Use #if's in the test code to remove failing tests. >>> >>> 2. Use [Category] attributes on the tests to declare which tests >>> shouldn't be executed. The Categories tab within NUnit allows you to >>> specify which categories are executed. >>> >>> The problem with (1) is a gigantic increase in code complexity, as each >>> vendor will have a different set of failing unit tests, so we'd potentially >>> need checks for every vendor on each method. This is incredibly bad. >>> >>> (2) would at least avoid the "line noise" implied by #if, though it >>> could also get very "busy" (with upwards of 7 [Category] attributes on a >>> given method). >>> >>> Thoughts? Alternatives? >>> >>> Thanks, >>> - Jon >>> >>> [0] OK, a third solution would be to actually fix all the errors so that >>> everything is green without using (1) or (2) above, but I don't think that >>> this is practical in the short term. >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "DbLinq" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/dblinq?hl=en -~----------~----~----~----~------~----~------~--~---
