MbUnit Re: Mutiple asserts in a single fixture

Wed, 04 Oct 2006 13:16:29 -0700

Hi,
 
Thanks for all your feedback.
 
Andy
 
On 10/4/06, Phil Haack <[EMAIL PROTECTED]> wrote:

The main issue is the following, using the example you gave:

 

    Assert.AreEqual("5", foo.X);

    Assert.AreEqual(42, foo.Y);

 

Suppose that foo.Y depends on the value of foo.X.  The first assertion failing might make the second one pass, when it shouldn't.


Multiple asserts in a test isn't the issue so much as the fact that as soon as one assert within the test fails, it may invalidate the results of the rest of the assertions in the same test, thus executing them could report false information.

 

Phil

 

From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Jeff Brown
Sent: Wednesday, October 04, 2006 11:10 AM
To: [email protected]; [EMAIL PROTECTED]


Subject: MbUnit Re: Mutiple asserts in a single fixture

 

Well certainly I'd be unwilling to give up multiple assertions per unit test.  Not all tests are so simple that there is exactly one value being verified.  When you're asserting multiple values within a test you're usually doing one of the following:

 

1. Verifying the sanity of the test fixture.

2. Verifying some compound state that involved several different values.

3. Verifying the extended lifecycle by performing a sequence of operations upon the object under test and verifying each one.

 

Of these, the only case I'd be concerned about is #3 but in practice it's not much of a problem.  There is no way to verify the lifecycle of the object under test without actually making it go through its transitions.  So you might as well have assertions after each transition to check.  You could decompose the test such that each transition is in its own method and we chain them together but IMNSHO that's often a waste of time.  A compiler could perform this transformation for you if you really care that much.  Sure the test can become more complicated but that shouldn't be surprising given its objective.  As per usual the appropriate response is to document the test more clearly, use sensible messages when reporting assertion failures and try to keep the test focused on one particular scenario.

 

Also note that not everyone uses MbUnit for unit testing!  Our Quality Engineering department uses a web testing framework built on top of MbUnit.  The one assertion per test rule would be completely inappropriate for them.

 

I think this last point is important so I'm going to repeat it again.  MbUnit is used for more than just unit testing!  The framework should provide affordances for other kinds of testing modalities.  As hammer's go, it's a very nice one...

 

 

I've actually been thinking about ways the framework could be opened up further to enable more intricate reporting scenarios.  For instance, our QE team would like to include periodic screenshots throughout the execution of their web tests.  There's no way to add this information to the MbUnit report short of creating a custom test runner.  Likewise, it should be possible to integrate monitoring of performance counters and other system resources and include them as out-of-band information in a report.

 

I would see no harm in a multi-assert facility being built that did something similar by recording the additional failures out-of-band to be included in the report rather than throwing assertions.

 

Here's one clean way to do it.  Create a block within which you are creating a "composite" assertion.  A composite assertion is simply a block of code, all of whose assertions will be allowed to run because AssertionExceptions will not be thrown.  Their results will be logged instead.  At the end of the block, we check whether a failure was noted and if so we raise a failure of our own.  So composite assertions could be nested.

 

DataStructure foo = objectUnderTest.DoSomething();

 

using (Assert.Composite("Returned data structure contains all expected values"))

{

    Assert.AreEqual("5", foo.X);

    Assert.AreEqual(42, foo.Y);

    CollectionAssert.AreEqual(new int[] { 1, 2, 3, 4 }, foo.Z);

}

 

Another possibility is something like Assert.Negate() though I'd probably avoid it.  We could of course do Assert.ExpectedException using an anonymous delegate.

 

 

For QE's reporting woes, I'd want to have some means of recording additional information to include in reports.  Could be used here too.  The basic idea is to make it possible for a test to append structured information to the report.

 

Create something like an AssertionHandler to capture what happens when an assertion passes or fails.  The default when an assertion fails is to throw an exception and do nothing when an assertion passes.  When inside a Composite assert block, we redirect assertion failure handling to do something different.  It could call a reporting method to report the failure instead of throwing.

 

We could use those same reporting methods too:

 

Report.Add(new AssertionFailureRecord("My assertion failed."));

Report.Add(new AssertionPassRecord("My assertion passed."));

 

And for our QE department, I could do something like this deep in the guts of my web test framework:

 

Report.Add(new WebPageSnapshot("some html", anImage));

 

A comparison assertion could record the exact strings used rather than just some formatted representation:

 

public void AreEqual(object expectedValue, object actualValue)

{

    Report.Add(new AssertionFailureRecord(String.Format(....), new EqualityAssertion(expectedValue, actualValue));

}

 

Notice the power we gain here by being able to record structured information in a report.  We could use it to provide better affordances in the test runner because there's extra meta-data in the report regarding each assertion (and other things besides that)...

 

Jeff.

 

From: [email protected] [mailto: [email protected]] On Behalf Of Phil Haack
Sent: Wednesday, October 04, 2006 9:45 AM
To: [EMAIL PROTECTED]; [email protected]
Subject: MbUnit Re: Mutiple asserts in a single fixture

I think Roy Osherove has a good argument against this.

 

http://weblogs.asp.net/rosherove/archive/2006/10/04/Avoid-multiple-asserts-in-a-single-unit-test_3A00_-revisited.aspx

 

Phil

 

From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Andrew Stopford
Sent: Wednesday, October 04, 2006 2:12 AM
To: [EMAIL PROTECTED]; [email protected]
Subject: MbUnit Mutiple asserts in a single fixture

 

Hi,

 

I wanted to raise an idea from James Avery about multiple asserts per fixture with addtional attribute and assert markers.

 

 

I have suggested to James that combintional test could help where your looking for mutiple inputs, however Jamies idea is to take several asserts in one fixture to let you conduct several tests. It's an interesting idea but one I am not sure about in that mbunit like other frameworks stops at the point of failure so that you fix that before re-running your tests again and that makes sense to me. Jame's idea is that if you get a fail then if the marker allows it you carry on but then fail the test when the run completes (and report the fail(s). I'd welcome your thoughts on this, would it be useful, does it make sense?

 

Andy

 





--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "MbUnit.User" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/MbUnitUser
-~----------~----~----~----~------~----~------~--~---

Reply via email to