|
You note, that it _may_ invalidate the results. It
may just as well not. My gut reaction to the proposal was much like yours
initially.
So I started thinking. Quite often when I have
assertions over some compound state I don't get to see enough information in the
report. It only shows me where part of the state diverged from what was
expected. So I end up having to use the debugger.
Bother.
On reflection I believe that a controlled facility
for compound assertions can be of great
value. So keep in mind what I proposed. The test writer
can explicitly declare blocks within which multiple assertions will be allowed
to run. So if the test writer wants to shoot himself in the foot he
can.
Thinking ahead a bit, I believe a mechanism like this
can be used to support extended reporting facilities because if we can intercept
assertion handling behavior we can log more details about them. So
building support for this feature into the core can enable other novel
extensions.
Jeff. From: [email protected] [mailto:[EMAIL PROTECTED] On Behalf Of Phil Haack Sent: Wednesday, October 04, 2006 12:12 PM To: [EMAIL PROTECTED]; [email protected] Subject: MbUnit Re: Mutiple asserts in a single fixture The main issue is the
following, using the example you gave:
Assert.AreEqual("5", foo.X);
Assert.AreEqual(42, foo.Y); Suppose that foo.Y
depends on the value of foo.X. The first assertion failing might make the
second one pass, when it shouldn’t.
Phil From:
[EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff Brown Well certainly I'd be
unwilling to give up multiple assertions per unit test. Not all tests are
so simple that there is exactly one value being verified. When you're
asserting multiple values within a test you're usually doing one of the
following: 1. Verifying the sanity
of the test fixture. 2. Verifying some
compound state that involved several different
values. 3. Verifying the
extended lifecycle by performing a sequence of operations upon the object
under test and verifying each one. Of these, the only case
I'd be concerned about is #3 but in practice it's not much of a problem.
There is no way to verify the lifecycle of the object under test without
actually making it go through its transitions. So you might as well have
assertions after each transition to check. You could decompose the test
such that each transition is in its own method and we chain them together but
IMNSHO that's often a waste of time. A compiler could perform this
transformation for you if you really care that much. Sure the test can
become more complicated but that shouldn't be surprising given its
objective. As per usual the appropriate response is to document the test
more clearly, use sensible messages when reporting assertion failures and
try to keep the test focused on one particular
scenario. Also note that
not everyone uses MbUnit for unit testing! Our Quality Engineering
department uses a web testing framework built on top of MbUnit. The one
assertion per test rule would be completely inappropriate for
them. I think this last point
is important so I'm going to repeat it again. MbUnit is used for more than
just unit testing! The framework should provide affordances for other
kinds of testing modalities. As hammer's go, it's a very nice
one... I've actually been
thinking about ways the framework could be opened up further to enable more
intricate reporting scenarios. For instance, our QE team would like to
include periodic screenshots throughout the execution of their web tests.
There's no way to add this information to the MbUnit report short of creating a
custom test runner. Likewise, it should be possible to integrate
monitoring of performance counters and other system resources and include them
as out-of-band information in a report. I would see no harm in
a multi-assert facility being built that did something similar by recording the
additional failures out-of-band to be included in the report rather than
throwing assertions. Here's one clean way to
do it. Create a block within which you are creating a "composite"
assertion. A composite assertion is simply a block of code, all of whose
assertions will be allowed to run because AssertionExceptions will not be
thrown. Their results will be logged instead. At the end of the
block, we check whether a failure was noted and if so we raise a failure of our
own. So composite assertions could be nested. DataStructure foo =
objectUnderTest.DoSomething(); using
(Assert.Composite("Returned data structure contains all expected
values")) {
Assert.AreEqual("5", foo.X);
Assert.AreEqual(42, foo.Y);
CollectionAssert.AreEqual(new int[] { 1, 2, 3, 4 },
foo.Z); } Another possibility is
something like Assert.Negate() though I'd probably avoid it. We could of
course do Assert.ExpectedException using an anonymous
delegate. For QE's reporting
woes, I'd want to have some means of recording additional information to include
in reports. Could be used here too. The basic idea is to make it
possible for a test to append structured information to the
report. Create something like
an AssertionHandler to capture what happens when an assertion passes or
fails. The default when an assertion fails is to throw an exception and do
nothing when an assertion passes. When inside a Composite assert block, we
redirect assertion failure handling to do something different. It could
call a reporting method to report the failure instead of
throwing. We could use those same
reporting methods too: Report.Add(new
AssertionFailureRecord("My assertion failed.")); Report.Add(new
AssertionPassRecord("My assertion passed.")); And for our QE
department, I could do something like this deep in the guts of my web test
framework: Report.Add(new
WebPageSnapshot("some html", anImage)); A comparison
assertion could record the exact strings used rather than just some formatted
representation: public void
AreEqual(object expectedValue, object actualValue) {
Report.Add(new AssertionFailureRecord(String.Format(....), new
EqualityAssertion(expectedValue, actualValue)); } Notice the power we
gain here by being able to record structured information in a report. We
could use it to provide better affordances in the test runner because there's
extra meta-data in the report regarding each assertion (and other things besides
that)... Jeff. From:
[email protected] [mailto:[EMAIL PROTECTED] On Behalf Of Phil Haack I think Roy Osherove
has a good argument against this. Phil From:
[EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Andrew Stopford Hi, I wanted to raise an idea from James Avery about
multiple asserts per fixture with addtional attribute and assert
markers. I have suggested to James that combintional test could
help where your looking for mutiple inputs, however Jamies idea is to take
several asserts in one fixture to let you conduct several tests. It's an
interesting idea but one I am not sure about in that mbunit like other
frameworks stops at the point of failure so that you fix that before re-running
your tests again and that makes sense to me. Jame's idea is that if you get a
fail then if the marker allows it you carry on but then fail the test when the
run completes (and report the fail(s). I'd welcome your thoughts on this, would
it be useful, does it make sense? Andy --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "MbUnit.User" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/MbUnitUser -~----------~----~----~----~------~----~------~--~--- |
- MbUnit Re: Mutiple asserts in a single fixture Jeff Brown
- MbUnit Re: Mutiple asserts in a single fixture Phil Haack
- MbUnit Re: Mutiple asserts in a single fixture Andrew Stopford
- MbUnit Re: Mutiple asserts in a single fixture Jeff Brown
