----- Original Message ----- From: "Noel J. Bergman" <[EMAIL PROTECTED]>
> It seems to me that we ought to adopt a more formal > methodology, develop a test strategy, and base the initial majority of > scripts on testing for compliance with the RFCs. Agreed, we should talk methodology and right set of tools. Here are some ideas on methodology. - Create protocol compliance verification tests using record replay scripts - Create Test suites to for end to end testing. This may involve multiple protocol tests. - Create tests to expose and verify specific problems. For instance Danny's end to end test crashes James unless proposed scheduler implemenation or watchdog build is used. There could be simulate thread pool problem where threads were not being retuned back. I have been trying to build some tools that will facilitate testing. These are the 2 types of general tools. - Protocol Simulation given a template file - JUnit Test Sequence and test suite assembly. It could be used to do something like send a mail, wait for mail to show up via POP3 or IMAP by writing separate tests for SMTP/POP3/IMAP and combining them together in a declarative manner. > I have looked at some of the recent posts, and get the sense that the test > scripts are ad hoc. Test tools etc. are a bit ad hoc. I thought about various alternatives but it seemed easiest to write something like ProtocolSimulator. It seemed to simple and none of the programs I know of seem to fit the bill. One good thing is that we have some tests and tools and have now started talking more about testing. > I am also of a mind that > we should look at using BSF as a key component of our testing framework. > That would let us leverage simple templates as well as more complex scripts, > including those adapted from other suites. Harmeet mentioned Python, which > can be used with BSF: http://www.lonsteins.com/articles/jython-bsf.html. > Done properly, this becomes a very useful general purpose test bed, > customized for a given protocol by additional beans and scripts. I think Jython is really useful. - It support the Python libraries. Good client libraries for protocols like POP3, SMTP, IMAP, NNTP etc. For example commons does not seem to have IMAP library. Writing non record-replay tests may be painful. - Easy to extend. - Very cool language and tool. Much easier to work with than Java. - Seamless mapping and integration with Java language. Most other Java languages like TCL, etc. have more convoluted mapping. One downside to any scripting language is the lack of strong typing. Compiling the code can be useful esp. if tests once created and working may not be touched for a long time at least not all code paths. It may be simpler and lesser learning curve to directly use Jython instead of Jython+BSF but either alternative works for me. Harmeet -- To unsubscribe, e-mail: <mailto:james-dev-unsubscribe@;jakarta.apache.org> For additional commands, e-mail: <mailto:james-dev-help@;jakarta.apache.org>
