2. I don't think Icarus shows tests in [DynamicTestFactory] either but it
does show tests in [StaticTestFactory].
Contract verifiers are pretty much just a different syntax for
[DynamicTestFactory] right now. We could provide the option for them to
work like [StaticTestFactory] too but keep in mind that will adopt the same
advantages limitations. For example, you can't use [StaticTestFactory] in a
generic class and the factory method has to be static... however you can see
the tests in Icarus and run them individually.
On reflection, we could make the determination be automatic. I guess if you
define the contract verifier in a static field then we could construct the
tests statically, otherwise we would make it them dynamic.
(For that matter, we could combine DynamicTestFactory and StaticTestFactory
into just TestFactory and use the same logic.)
3. Why not rewrite all of the tests that you want to reuse in the form of a
contract verifier? The big advantage of a contract verifier is that you can
add properties to them that control their structure. The idea is that the
contract verifier can itself decide which tests to include and which to
exclude.
Right now a lot of the problems seem to come about because of the mechanism
that you are using for reuse. Ordinary test fixtures just aren't very good
at reuse. Test factories and contract verifiers are better.
public class EntityContract<T> : AbstractContract
where T : Entity
{
private readonly string dataSourceName;
public EntityContract(string dataSourceName)
{
this.dataSourceName = dataSourceName;
// or other required parameters
}
// optional parameters
public bool EnableSerializationTests { get; set; }
public bool EnableFancyFeaturesTests { get; set; }
public bool EnableIntegrationTests { get; set; }
public override IEnumerable<Test> GetContractVerificationTests()
{
yield return GenerateBasicTest1();
yield return GenerateBasicTest2();
if (EnableSerializationTests)
{
yield return GenerateSerializationTest1();
yield return GenerateSerializationTest2();
}
// ...
}
// ...
}
In other words, I am suggesting that put ALL of your explicitly reusable
testing logic into contract verifiers instead of trying to parse out test
metadata and regenerate tests dynamically. Basically, if the tests are
going to be reused by others then I think it's worth the extra effort of
implementing them in a reusable way up-front instead of trying to shim them
into a reusable wrapper.
Jeff.
_____
From: [email protected] [mailto:[email protected]] On
Behalf Of Mark Kharitonov
Sent: Friday, September 04, 2009 11:59 AM
To: [email protected]
Subject: MbUnit Re: How to extract the test Metadata from within the SetUp
method?
2. If a contract verifier consists of 20 individual test cases, I cannot run
a single test case, in order to debug just the failing code. I have to run
all the test cases. This is all fine, if the test cases are light, but what
if they are not? Why a contract verifier is treated differently then a
dynamic test fixture? I really want to be able to refer to individual
contract verifier test cases.
To summarize, it is my personal opinion that contract verifier must be
expandable in Icarus, so that individual test cases could be run and
debugged (of course, that means these individual test cases should be
runnable from Echo and Gallio task). I think we can do fine with only the
native Gallio tools supporting it. In fact, I stopped using TD.NET and work
exclusively with Icarus + VS, because Icarus lets me debug test selection,
regardless whether they are statically compiled or dynamic.
3. http://code.google.com/p/mb-unit/issues/detail?id=521
Nested loops is not a very scalable solution. In my scenario dynamic tests
wrap existing statically compiled ones, binding the first argument to some
known value.
If the prototype test is combinatorial, the respective dynamic test should
be combinatorial as well. How can I do it with nested loops? Don't think I
can.
Jeff, I agree with completely on striving to simplicity. However, as
Einstein once said "make things as simple as possible, but not simpler". My
scenarios seem complex, but I have complex testing needs:
* I have to unit test (actually, they are integration tests, but never mind)
the base entity infrastructure.
* Actual business entities should be tested to make sure they do not break
the base infrastructure. These tests come in addition to the specific
business tests for the respective entity, which do not concern me.
* The business entity developer cannot be expected to copy/paste the base
infrastructure tests and manually eliminate those, which test infrastructure
features not supported by the respective business entity. This is because,
the base infra unit tests are added all the time and the developer will be
constantly out of sync, let alone other problems always associated with
copy/paste.
* A business entity may not support some features, provided by the
infrastructure. So, naturally, not all the base infra unit tests are
relevant for it, but most are indeed.
These testing needs define my solution, I did not come with it out of fancy.
It was an evolution, from simple mstest to mstest with in-memory data
provider and custom combinatorial test logic to simulate combinatorial tests
and finally to MbUnit, because I had no choice. Making mstest behave like
MbUnit is just too costly.
BTW, [Disable] attribute may be a good idea in general, but it will do no
good to me, because I place all the logic of creating unit tests to make
sure business entities do not break the infrastructure in a single generic
base class, which serves more than one business entity type. So it must
suppress unwanted tests dynamically based on the features supported by the
entity under test and the features expected by the particular unit test. The
attribute is of no help here.
On 04/09/2009, at 10:32, Jeff Brown wrote:
1. Debugging contract verifiers is the same as anything else. Stick a
breakpoint within the code of the contract verifier that you want to
hit. That said, they are intended to be black boxes for the most part
in the same way that assertions are. Users of contract verifiers
aren't really expected to set breakpoints in them (but they can).
One implication is that you can't set a breakpoint on the field that
integrates the contract verifier into the fixture. You have to set a
breakpoint within the code of the verifier itself. There isn't
anything that we can do about this.
2. This is not supported. Think of contract verifiers as super-fancy
asserts. I know it is inconvenient but given the dynamic nature of
the contract verifier it's the best we can do. In general test
runners are not able to obtain information about the individual tests
within the contract verifier until they run. Consequently there's no
way to pick them or filter them.
This is an area that may change later on. Gallio has a little bit of
code here and there that could ultimately be used to enable a test
runner to inspect and manipulate dynamic test cases. Truth be told
though, I don't expect much to happen here mainly for usability
reasons. We could perhaps support some of this functionality in
Icarus, Echo, Sail and other native Gallio runners but TDNet,
ReSharper and VSTT are unlikely to be able to support these features
without some significant UI work.
3. Yes that's true. Please feel free to open up a feature request for
MbUnit extensions to support data-driven custom test suites. Sounds
like something Yann might enjoy designing. I'll be honest though,
you're probably much better off with some nested for-loops than any
additional API affordances we may provide for this case.
4. It is not possible to override a single contract verifier test
case. However contract verifiers support extensive customization via
properties. You can also subclass them if you like to add new
features (although the built-in ones aren't really designed with
subclassing in mind yet.)
General comment:
I have noticed that several of the things you are trying to do with
MbUnit are very complex. For example, it sounds like you use abstract
test fixtures very heavily and you are using many other dynamic
features. While MbUnit v3 is one of the most powerful .Net test
frameworks currently available, I do recommend opting for simplicity
when possible.
I have personally written and experienced a lot of pain from writing
very clever test fixtures with fancy setup/teardown methods, template
methods, and fancy extensions. In the beginning it's pretty cool to
see just how much test code can be condensed by factoring out common
bits in novel ways but later on it can become problematic. Often the
complex abstractions are almost but not quite perfect which results in
the need to add shims here and there during reason to disable some
tests and tweak other behaviors.
(But just for you I'm planning to add a [Disable] attribute to
indicate that a class / method is not a test. Be careful with it
though.)
I think that Brad and Jim are on the right track with xUnit.net.
Granted, there are some things that you cannot do as easily with
xUnit.net but I believe the built-in constraints do help to stay
focused on what matters: developing robust and cohesive test suites.
That said, it is a lot of fun for me as a framework author to have
someone like you pushing the edges of the framework and finding
opportunities to make it better!
Cheers,
Jeff.
On Aug 27, 12:49 am, Mark Kharitonov <[email protected]>
wrote:
Another question - is it possible to override a single contract test
case?
Thanks.
On Aug 27, 9:42 am, Mark Kharitonov <[email protected]> wrote:
I have played a bit with it.
A few things I do not understand about contract verifiers:
1. How to debug them?
2. How to run just one contract test case? Icarus shows the contract
verifier tests as one atomic unit.
3. If a contract verifier should include a combinatorial test, then
one has to implement it all by oneself, because dynamic test
generation does not provide this service, unlike the statically
compiled unit test cases, am I correct?
Thanks.
On Aug 24, 10:04 pm, "Jeff Brown" <[email protected]> wrote:
That's pretty much it but you could instead create a TestSuite that contains
multiple TestCases.
There is a bunch of framework code for data-driven testing in
Gallio.Framework.Data. Some of this may be useful to use however I expect
you will find it easier to keep it simple and generate combinatorial tests
using nested loops.
Jeff.
_____
From: [email protected] [mailto:[email protected]] On
Behalf Of mark Kharitonov
Sent: Monday, August 24, 2009 9:05 AM
To: [email protected]
Subject: MbUnit Re: How to extract the test Metadata from within the SetUp
method?
On the subject of manually writing combinatorial tests.
I have a question here.
A combinatorial test appears just once in Gallio.Icarus. It serves as a well
defined report unit, under which the actual test steps appear.
How can I achieve the same effect with manually created combinatorial tests?
Looks like I need to create a test and then the test steps as children of
that test, is that right?
BTW, Is it possible to reuse Gallio code when manually creating
combinatorial test steps?
Thanks,
On Sat, Aug 22, 2009 at 10:37 PM, Jeff Brown <[email protected]> wrote:
At runtime you can obtain information about the current test using
TestContext.Current.Test.
What you cannot do at the moment is to easily obtain information about other
tests. Well, in principle the whole test model is fully accessible but in
practice you'll find it to be inconvenient to work with.
I have been thinking about what you said with respect to regenerating
dynamic test cases from static ones for testing entities of different types.
I think I would recommend that you instead implement a new contract
verifier.
Take a look at IContract, VerifyContractAttribute and some of the existing
contract verifiers like EqualityContract.
You could pretty easily implement an EntityContract of your own. Basically
a contract generates test cases in the same manner as a [StaticTestFactory]
but they are packaged in a more convenient form for reuse.
However, contracts then have the same disadvantage as the static & dynamic
test factories in that they do not (yet) provide convenient abstractions for
representing data-driven test cases. I expect that this can be remedied
with a little work but it's out of scope for v3.1. As a result you will
have to write your own combinatorial test generators and such but it's
pretty trivial code.
Anyways here's what an entity contract might look like to the user:
[VerifyContract]
public readonly EntityContract MyEntityContract = new
EntityContract<MyEntity>(... options...);
Jeff.
-----Original Message-----
From: [email protected] [mailto:[email protected]] On
Behalf Of Mark Kharitonov
Sent: Saturday, August 22, 2009 11:52 AM
To: MbUnit.User
Subject: MbUnit How to extract the test Metadata from within the SetUp
method?
I mean, besides using the reflection on the test method by its name. I
expected to find something like GetMetadata method on the Test class, but
there is none.
--
Be well and prosper.
==============================
"There are two kinds of people.Those whose guns are loaded and those who
dig."
("The good, the bad and the ugly")
So let us drink for our guns always be loaded.
==========================================================================
There are two kinds of people. Those whose guns are loaded and those who
dig.
(The good, the bad and the ugly).
So let us raise our cups for our guns always be loaded.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"MbUnit.User" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/mbunituser?hl=en
-~----------~----~----~----~------~----~------~--~---
