At 02:27 PM 6/6/2005 -0700, John Anderson wrote:
It also seems like our tests would be more valid if we were using real Twisted code instead of something else.
That would be a true statement if we were only talking about integration tests. However, there is a difference between unit testing and integration testing. Unit testing tests components in isolation, integration testing tests them with other components. Unit testing promotes development speed, architectural soundness, and developer confidence, while integration testing supports quality assurance for code release. Unit tests should be very, very fast, preferably with centisecond runtimes or better for individual tests. Integration tests can be slower, although being fast is always good.
Unit tests should always, always, always succeed, with never a broken checkin. Integration tests, if they are also being used as acceptance tests, can be allowed to fail, because acceptance tests are used to show the degree of feature-completeness of a product, and in that case they simply reflect unfinished features.
As a general rule of thumb, unit tests should be fast enough that one does not feel burdened by running them after *every line or two* of new code written, while integration tests should be comfortably run once a day, or overnight.
Chandler currently has relatively few tests that qualify as unit tests; it needs more. Chandler already has lots of integration tests, but they are no substitute for actual unit tests.
Now, even if we are talking about integration tests, note that Brian's tests use a "string transport", which is not "real" socket code, so by your argument this would not be as "valid" as using sockets. But, if you say that StringTransport is "real Twisted code" because it comes with Twisted, then all we need to do is contribute a usable mock reactor and replacement fixtures to the Twisted code base, at which point it will become "real Twisted code". :)
A major idea in the concept of unit tests is to test things in isolation, minimizing the number of external points of failure (such as external servers, connectivity, etc.) for the test. Mock objects and stubs are a typical practice for supporting unit tests of components with external dependencies. The use of interfaces to define what is expected on each side of such dependencies is a supporting practice.
The last time I did any significant work with Twisted, it had something like 6 different reactor implementations and variations thereof. A mock reactor would just be #7, or #8, or how many ever such implementations they have now. Twisted is built on interfaces, which means that there are behaviors that reactors are expected to provide and conform to, and which all other components of Twisted can expect, and whose limits they may not exceed. This means that it is a bug in Twisted if any part of Twisted does not function correctly with implementation #N of the IReactor* interfaces.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Open Source Applications Foundation "Dev" mailing list http://lists.osafoundation.org/mailman/listinfo/dev
