One thing that might make this process easier for regression testing things like loading of older file formats, is the use of approval tests. http://approvaltests.sf.net
It's similar to the the way JUnit asserts work, but you can use it to compare screenshots of an app in addition to comparing output files, and values. Let's say that we want to compare ArgoUML version 0.27 with 0.28.beta something. It takes a screenshot after loading a project file. You "approve" the screenshot. You then run the same test using 0.28 beta. It automatically takes the screenshot of the new app and compares the two. If they match, it passes. You can also use it to compare different runs of the "save" operation, and compare the output file from 0.27 and 0.28 beta. This blog shows some examples of how to use it: http://blog.approvaltests.com/ Hope this helps, Mark On Fri, Mar 20, 2009 at 10:57 AM, Dave Thompson <[email protected]> wrote: > Linus Tolke wrote: > > I have now created the wiki page > > http://argouml.tigris.org/wiki/Test_Report_0.28 more or less after what > > my vision of your mail Dave. Was it also what you were thinking? > > This looks great. I will work on this to improve it. The main thing will > be > to point beta testers to it, and I hope that it can be made fairly self > explanatory. We'll try it and see how it goes; if it doesn't work well, > then > it's no big loss. > > > 2009/3/20 Tom Morris <[email protected] <mailto:[email protected] > >> > > My personal opinion is that any approach based on manual testing, > > particularly manual testing by developers, is doomed to failure. > > > > The way to get adequate human coverage is to have a large user > > population and to engage them effectively in beta testing. You > > don't > > need 100% coverage. If users aren't using something, it doesn't > > matter whether it works or not. > > This is making one big assumption that feature value is proportional to > frequency of use. This is not always the case, imagine the 'Nuclear > Reactor > Emergency Shutdown' feature, or the driver airbag in a car! I'm being > silly, > but I do think that there will be some ArgoUML features that are not used > much, yet are pivotal to some particular users. > > For example, the loading up of models saved with old versions of ArgoUML. > Most users probably won't test this much, but if you are a user with an > archive of lots of old models which need to be viewed from time to time, > then > it will be very important to you. Some features can be tested > automatically, > others are very hard. Does an old file *look* right after it is loaded? > The > model may be right, but is it legible on the screen? This is not so easy > to > test automatically. > > I think we are in agreement that lots of testing (automatic or manual) is > good, and getting users to do beta testing is worthwhile. While we don't > have any incentives for beta testers, I think the most we can realistically > expect for the average beta tester is that they will play with the beta > versions, and possibly use them for their day to day work. > > The wiki list will hopefully give those beta testers who want to go a > little > further, some guidance on what sort of things they might want to test. It > could be used as a focus point to help users shorten beta periods if they > want to see new releases more quickly. It might even provide an easy way > for > some people just to get involved that little bit more, which is a good > thing > considering our general lack of resource. > > We'll see how it goes. > > Dave > > ------------------------------------------------------ > > http://argouml.tigris.org/ds/viewMessage.do?dsForumId=450&dsMessageId=1366282 > > To unsubscribe from this discussion, e-mail: [ > [email protected]]. > To be allowed to post to the list contact the mailing list moderator, > email: [[email protected]] > -- Mark Fortner blog: http://feeds.feedburner.com/jroller/ideafactory ------------------------------------------------------ http://argouml.tigris.org/ds/viewMessage.do?dsForumId=450&dsMessageId=1366373 To unsubscribe from this discussion, e-mail: [[email protected]]. To be allowed to post to the list contact the mailing list moderator, email: [[email protected]]
