Craig Berry coined an interesting phrase for what I've been doing over on vmsperl. http:[EMAIL PROTECTED]/msg05063.html
"Bull In A China Shop Testing" Take a knowledgeable tester/programmer/user and place him in an environment of which he knows very little (such as plopping a Unix guy onto VMS). Then set him free to try and make the thing work just by reading the published directions. What ensues is typically a flurry of questions, often beginning with "Why". "Why does it work this way?" "Why am I not allowed to do X?" "Why can't I do Y?" "Why do I have to do X, Y and Z when I only have to do X on Unix?" "Why doesn't X work?" (This stage could be called "Five Year Old Kid Testing") Some will be questions that would be obvious to a normal user. Those can be safely discarded. Others will be questions that wouldn't be so obvious. Those are candidates for expanding the FAQ and docs. Others will expose documentation that is either out of date, wrong or makes assumptions about the user's environment. Be patient with the Bull and you can improve your docs wonderfully. A project who's test suite has "expected failures", they've lost confidence in the tests because they always seem to fail for things that aren't really bugs, is a perfect candidate for a Bull. One of the first questions will be "Why are these tests failing?" People more familiar with the project will have unconsciously trained themselves to step lightly around certain areas of the code that they know are fragile or messy. There's always certain spots that everyone knows is a problem, but nobody really wants to go in and clean up. The Bull, being a bull, does not step lightly. He'll stomp around shaking things, looking at messy code that everyone else has written off and loudly declaring "Wow, what a mess!" [1] Asking uncomfortable questions can often lead to the problems being solved. Sometimes a little more incentive is needed, like the Bull trying to fix the problem himself, often breaking lots of things in the process motivating everyone else to come in and clean up the mess (and hopefully fixing the original problem in the first place). So the basic idea is to introduce an unfamiliar, yet otherwise knowledgeable, person to a stagnant project. The Bull. He asks lots of troublesome questions and makes fumbled attempts at fixing problems (fumbled because of the unfamiliar environment). The net result is a lot of assumptions get questioned, confidence in the existing docs and tests shaken, nasty code gets dusted off and in general breathes some new life in the project. Sort of a Darwinian approach to QA. Can you build bull-proof China shop? The downside being that you've got this big oaf charging around breaking stuff in the middle of your project. Essentially, a walking proof of Brooks' Law. In order for it to work, the Bull must be able to learn the new environment fast (*without* losing his basic naivete), rapidly correct his mistakes, work well with others, have a sense of humor about all the criticism and friction which has been caused. It helps immeasurably if the Bull can contribute useful code once in a while, especially when it comes to solving nasty, old problems. People will forgive a lot of someone that writes good code. The proportion of good patches to flubbed patches should rise rapidly over time as the Bull gets familiar with the environment. This is why its important that the part of the Bull be played by someone who is otherwise experienced. [1] Add decorum as necessary. -- Michael G. Schwern <[EMAIL PROTECTED]> http://www.pobox.com/~schwern/ Perl Quality Assurance <[EMAIL PROTECTED]> Kwalitee Is Job One A tickle of paste Teases down my sweaty thigh Oh, plug it again! -- ignatz
