On Thu, 19 Nov 2009 17:16:02 +0000 Paul Cockings <ds...@cytringan.co.uk> wrote:
> Stevan Bajić wrote: > > Paul is hot getting 3.9.0 out. :) > > btw: When do we consider 3.9.0 to be finished? What has to be meet for > > releasing 3.9.0? > > > > > We need some help making some test procedures. We have lots of VM's > running, and some funky hardware, but what are we going to test? > > Dspam is so flexible its really hard to test! Which MTA's, which DB's? > > The test right now is to check its compiling on various platforms. I > still don't have a clear idea how we check the 'quality'. > I guess we need to start with a large corpus? > Quality is something that can mean different things to different users. If I try to define quality in relation to DSPAM then I would define: 1) Quality of classification (one single message) Description: I send X to be processed by DSPAM and expect it to have result Y. Testcase: dspam --user xyz --classify --deliver=summary --stdout < X Success (if Y = Innocent): X-DSPAM-Result: xyz; result="Innocent"; class="Innocent"; probability=0.0031; confidence=0.50; signature=N/A Failure (if Y = Innocent): X-DSPAM-Result: xyz; result="Spam"; class="Spam"; probability=0.9315; confidence=0.50; signature=N/A Success (if Y = Spam): X-DSPAM-Result: xyz; result="Spam"; class="Spam"; probability=0.9315; confidence=0.50; signature=N/A Failure (if Y = Spam): X-DSPAM-Result: xyz; result="Innocent"; class="Innocent"; probability=0.0031; confidence=0.50; signature=N/A 2) Quality of tokenization (one single message) Description: I send X to be processed by DSPAM and expect tokens A, B, C to show up in my storage backend. [Y/N] Testcase: Would need to much time to write that down. But basically it involves processing a message and then use dspam_dump to check for the tokens. 3) Quality of binaries (their functionallity) Description: I use the DSPAM tool X and expect it to do action Y. [Y/N] Testcase (one of many. We have a lot of tools in place for DSPAM): I process a bunch of messages and then I run "dspam_clean -s0 username" to clean all signatures and then I check my storage backend to see if really all signatures have been purged. Testcase (one of many. We have a lot of tools in place for DSPAM): I process a binary file (for example /bin/ls) with DSPAM and expect DSPAM to: * Deliver either class Spam or Innocent * Not produce any output but log an error in the logs * I expect DSPAM to NOT crash 4) Quality of documentation Description: Check that everything from the documentation is clearly understandable and not claiming something that is not valid. 5) Quality of features (core features) Description: I use a official feature X from DSPAM 3.8.0 and expect it to work the same way in DSPAM 3.9.0 Testcase (one of many. We have a lot of features in place for DSPAM): Create a group file for DSPAM and add a "shared" group definition. Start the dspam binary and check in the logs if the defintion works as advertised. Testcase (one of many. We have a lot of features in place for DSPAM): Call dspam_admin to change a preference (if your storage backend supports it). Check the storage backend to see if the preference was changed. etc, etc, etc... I could go on and on and on. But right now I have to leave the house. Have an appointment. I think however that you get my point. Basically we need to define first what quality we want to mesure. Then we need to define the test cases. Then we need to define what we expect from those test cases (we could use DSPAM 3.8.0 as a base and say that each metric in DSPAM 3.9.0 should be at least on the same level as in 3.8.0). To make the whole thing short: We need to quantify and qualify our test. Then we can proceed. If we can't qualify nor quantify then we can't test. > The best quality we know so far is from those people running the Beta's > live... can we ever make a system to test 'offline' ? > - > YES! We can! -- Kind Regards from Switzerland, Stevan Bajić ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Dspam-devel mailing list Dspam-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspam-devel
