On Sat, May 3, 2008 at 1:43 PM, Noel J. Bergman <[EMAIL PROTECTED]> wrote: > Bernd Fondermann wrote: > > Noel J. Bergman wrote: > > > > The proposal is based on the fact that every message delivered to the > zone > > > will be disposable spam. Therefore, unlike performing some sort of faux > > > release without any basis, we will be testing in a risk-free > environment. > > > Every message can be dropped, the database can be corrupted, the server > can > > > leak memory and crash, and no one should care other than to fix it. > > > Then you are talking about a closed environment test, so do I. > > We're possibly differing on some English language semantics in terminology. > As long as we agree with the mechanics of the proposal, as above, I don't > care if you call it Nancy. :-) > > > > > As an example of why this sort of testing is the right thing to do, > rather > > > than the idiocy of pushing out releases without real-world testing, > consider > > > our current situation with SVN. > > > I never said we should release without testing beforehand. > > It is you who is ignoring the fact we have a safe black box > > testing tool right here in our project. > > No, you appear to be ignoring that blackbox testing using Postage in > isolated networks is nowhere near sufficient to provide any basis for > assurance of real-world functionality. I am addressing that lack. > > > > At the bottom line, I am happy that there seems to emerge some kind of > > common goal to start from TRUNK and put it into heavy testing, be it on > > a Solaris zone or locally using Postage. > > See above regarding Postage. And keep in mind that I've used Postal and > Rabid for isolated testing in the past, but it (too) is not sufficient. We > need the real world exposure.
Every test setup has its pros and cons. Unit test have a specific use, so have isolated, reproducible functional or load tests. Nancy (solaris zone, that is ;-)) has the disadvantage that it probably will reveil some bugs/problems which are not (easily) reproducible because we don't control the test data/load. This would make some people with proper knowledge of testing not call it a 'test', more an 'experiment'. I think it's worthwhile anyway, but you'd have to fall back to other tests after running into that certain family of seldom problems. I seem to have a hard time pitching Postage (which I actually wrote after - or better because of - working with Postal and Rabid), which allows to use specialized message factory objects. Write a custom factory for every specific mail which leads to a specific false behavior inside the Server. This makes Postage a fit for any kind a functional test you might (have to) come up with. Bernd --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
