Re: [U2] Software Test
Hi, we used an automated tester called Testpro a few years ago. You can set it up to run through your system using data in a file as input or you can load the script with the data. It works by recording what you do and capturing the screen output. It works..BUT it takes longer to maintain the scripts then do the changes. The overhead proved to much, we had about 30 developers at the time. We produce test plans for everything we do. It is the standard of the test plan and the quality of the person doing the testing that counts. Good luck Les Sherlock Hewkin Project Manager Group Financial Systems I.T. Department Ryehill House Ryehill Close, Lodge Way Industrial Estate, Northampton. NN5 7UA T 01604 592289 M 07917 856195 -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Lee Leitner Sent: 22 September 2009 23:25 To: U2 Users List Subject: Re: [U2] Software Test Bill: One strategy that improves testing results is to correctly define what a successful test is. A successful test is one that *finds* errors in the code. If no errors are found, then either there aren't any (and that's highly unlikely,) or the tests aren't good enough to trigger the problems. Ideally, you get a hold of someone who knows the application very well, and has a demonic streak. Have that person follow the strategy Ken Hall suggested in the earlier post. If they are immediately successful (i.e. finding errors,) track the rate of error discovery. When the rate falls below, say one per day, you might be at a cutoff point. Get a second opinion from another destructive expert if you can, particularly if the first fails (i.e. finds no errors.) Basically, nothing beats an application expert bent on breaking the software for finding bugs the fastest. What your boss is proposing to do is to kill several hundred bacteria by boiling the ocean. Managing test case coverage for a typical application is a huge task. And when you'll get a positive ROI is unclear: At what point will the test system have enough critical mass (test coverage) to actually demonstrably improve reliability and user satisfaction? Lee On Tue, 22 Sep 2009, Brutzman, Bill wrote: My boss has a new grand scheme for software testing. To test each program revision, he wants scripts to inject sample data into user apps and then another (big) program looks at EVERYTHING that comes out, comparing it to a baseline. To me, the scope of this project is huge, perhaps bigger that our ERP system. So far, I have been unable to talk him out of it. Thus, I am wondering what other people are doing in the way of software test. Virtually all of the source code that we have is in UniBasic. I do not know what jargon to use to describe the test that I do now. On a spreadsheet, I layout the various possiblities and then try each scenario... One-by-one... Fixing bugs as I find them and then starting the batch of tests from scratch. --Bill ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users -- Lee J. Leitner, Ph.D. leitn...@acm.org http://www.leitner.org/~leitnerl The world can only be grasped by action, not by contemplation. The hand is the cutting edge of the mind. -- Jacob Bronowski V.13.0 --- ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users html head meta http-equiv=Content-type content=text/html; charset=UTF-8 /head body P style=MARGIN-TOP: 0pt; MARGIN-BOTTOM: 0ptSPAN style=FONT-SIZE: 8.2pt; FONT-FAMILY: 'MS Sans Serif'This e-mail and any attachments are confidential and intended solely for the use of the addressee only. If you have received this message in error, you must not copy, distribute or disclose the contents; please notify the sender immediately and delete the message. /SPAN/P P style=MARGIN-TOP: 0pt; MARGIN-BOTTOM: 0ptSPAN style=FONT-SIZE: 8.2pt; FONT-FAMILY: 'MS Sans Serif'This message is attributed to the sender and may not necessarily reflect the view of Travis Perkins plc or its subsidiaries (Travis Perkins). Agreements binding Travis Perkins may not be concluded by means of e-mail communication. /SPAN/P P style=MARGIN-TOP: 0pt; MARGIN-BOTTOM: 0ptSPAN style=FONT-SIZE: 8.2pt; FONT-FAMILY: 'MS Sans Serif'E-mail transmissions are not secure and Travis Perkins accepts no responsibility for changes made to this message after it was sent. Whilst steps have been taken to ensure that this message is virus free, Travis Perkins accepts no liability for infection and recommends that you scan this e-mail and any attachments. /SPAN/P P style=MARGIN-TOP: 0pt; MARGIN-BOTTOM: 0ptSPAN style=FONT-SIZE: 8.2pt
Re: [U2] Software Test
Bill, We use scripts that run our application, but the big caveat is that we are 100% Web based so there are a lot of tools to do this. There are tons of open source applications like: MaxQ, Anteater, Bugkilla, Rational AppScan, and VNCRobot. You got to love those names. Anyway, you should be able to take any scripting language and feed the program and test the output. When we were telnet based, we had a person that was not familiar with the application to test it. She actually tele-commuted and would find a plethora of errors and submit them to our Bugzilla system. We would fix them and she would do her thing again. She was instrumental in writing our scripts for our web base programming. Now that we have all of these scripts we only need her to update them when we change or add prompts. Or when we write a new subsystem. Regards, Doug www.u2logic.com ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Software Test
Also take a look at Hudson, http://wiki.hudson-ci.org/display/HUDSON/Meet+Hudson. It can help with a lot of germane tasks for builds, testing...etc and it is easy to create customized plugins... Doug wrote: Bill, We use scripts that run our application, but the big caveat is that we are 100% Web based so there are a lot of tools to do this. There are tons of open source applications like: MaxQ, Anteater, Bugkilla, Rational AppScan, and VNCRobot. You got to love those names. Anyway, you should be able to take any scripting language and feed the program and test the output. When we were telnet based, we had a person that was not familiar with the application to test it. She actually tele-commuted and would find a plethora of errors and submit them to our Bugzilla system. We would fix them and she would do her thing again. She was instrumental in writing our scripts for our web base programming. Now that we have all of these scripts we only need her to update them when we change or add prompts. Or when we write a new subsystem. Regards, Doug www.u2logic.com ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Software Test
Bill - Years ago, while working at ADP Dealer Services, we were asked to attempt the same thing. Your take on the scope of the project is correct. It is a hugh project that requires some way to maintain the scripts in sync with the code as well as to store the sets of responses to enter. A top level team of ADP developers looked into the project and concluded that it was not cost effective. Due to the ever changing nature of the tests that are required for new apps and feature changes, keeping the scripts in sync with the next version of changes requires as much manpower as the manual testing. A better strategy for serious testing is to develop a separate staff that knows the application well and just is responsible for testing and development of test plans (an art in of itself that programmers are not very good at). I believe that this would offer a more cost effective strategy and get your company better testing that unit tests done by the programmers. In some programming environments (other than MV), there are products to do this kind of instrumented testing, but the only ones I ever investigated cost more that our entire budget for the QC staff that did our testing and a lot more at ADP. Ken At 12:57 PM 9/22/2009, you wrote: My boss has a new grand scheme for software testing. To test each program revision, he wants scripts to inject sample data into user apps and then another (big) program looks at EVERYTHING that comes out, comparing it to a baseline. To me, the scope of this project is huge, perhaps bigger that our ERP system. So far, I have been unable to talk him out of it. Thus, I am wondering what other people are doing in the way of software test. Virtually all of the source code that we have is in UniBasic. I do not know what jargon to use to describe the test that I do now. On a spreadsheet, I layout the various possiblities and then try each scenario... One-by-one... Fixing bugs as I find them and then starting the batch of tests from scratch. --Bill ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users Ken Hall www.old-scholls.com mailto:k...@old-scholls.com phone: 503-702-7841 ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Software Test
Ken: Thanks for writing. I neglected to mention that there is only one programmer here... me. --Bill -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ken Hall Sent: Tuesday, September 22, 2009 4:21 PM To: U2 Users List Subject: Re: [U2] Software Test Bill - Years ago, while working at ADP Dealer Services, we were asked to attempt the same thing. Your take on the scope of the project is correct. It is a hugh project that requires some way to maintain the scripts in sync with the code as well as to store the sets of responses to enter. A top level team of ADP developers looked into the project and concluded that it was not cost effective. Due to the ever changing nature of the tests that are required for new apps and feature changes, keeping the scripts in sync with the next version of changes requires as much manpower as the manual testing. A better strategy for serious testing is to develop a separate staff that knows the application well and just is responsible for testing and development of test plans (an art in of itself that programmers are not very good at). I believe that this would offer a more cost effective strategy and get your company better testing that unit tests done by the programmers. In some programming environments (other than MV), there are products to do this kind of instrumented testing, but the only ones I ever investigated cost more that our entire budget for the QC staff that did our testing and a lot more at ADP. Ken At 12:57 PM 9/22/2009, you wrote: My boss has a new grand scheme for software testing. To test each program revision, he wants scripts to inject sample data into user apps and then another (big) program looks at EVERYTHING that comes out, comparing it to a baseline. To me, the scope of this project is huge, perhaps bigger that our ERP system. So far, I have been unable to talk him out of it. Thus, I am wondering what other people are doing in the way of software test. Virtually all of the source code that we have is in UniBasic. I do not know what jargon to use to describe the test that I do now. On a spreadsheet, I layout the various possiblities and then try each scenario... One-by-one... Fixing bugs as I find them and then starting the batch of tests from scratch. --Bill ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users Ken Hall www.old-scholls.com mailto:k...@old-scholls.com phone: 503-702-7841 ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Software Test
I don't know of any tools for the MV world but I can give you some terms to look up to see how people do things in other environments. Test Driven Development http://en.wikipedia.org/wiki/Test_driven_development is common concept in Extreme Programming and in Agile Development. In a nutshell, you write the tests first and then work on the code until it passes the tests. A lot of people swear that doing it this way actually reduces the amount of time that it takes to deliver a project. I personally haven't stuck to it enough to say. http://en.wikipedia.org/wiki/Unit_test http://en.wikipedia.org/wiki/Agile_software_development http://en.wikipedia.org/wiki/Extreme_Programming In my Java code, I have a series of unit tests (using JUnit) that get run, automatically, every time I build my project. It's nice to know that, if I break something, I'm going to find out at build time rather than after it goes out the door. I should probably have a lot more of them. Hope it helps. -Ben On Tue, 2009-09-22 at 16:35 -0400, Brutzman, Bill wrote: Ken: Thanks for writing. I neglected to mention that there is only one programmer here... me. --Bill -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ken Hall Sent: Tuesday, September 22, 2009 4:21 PM To: U2 Users List Subject: Re: [U2] Software Test Bill - Years ago, while working at ADP Dealer Services, we were asked to attempt the same thing. Your take on the scope of the project is correct. It is a hugh project that requires some way to maintain the scripts in sync with the code as well as to store the sets of responses to enter. A top level team of ADP developers looked into the project and concluded that it was not cost effective. Due to the ever changing nature of the tests that are required for new apps and feature changes, keeping the scripts in sync with the next version of changes requires as much manpower as the manual testing. A better strategy for serious testing is to develop a separate staff that knows the application well and just is responsible for testing and development of test plans (an art in of itself that programmers are not very good at). I believe that this would offer a more cost effective strategy and get your company better testing that unit tests done by the programmers. In some programming environments (other than MV), there are products to do this kind of instrumented testing, but the only ones I ever investigated cost more that our entire budget for the QC staff that did our testing and a lot more at ADP. Ken At 12:57 PM 9/22/2009, you wrote: My boss has a new grand scheme for software testing. To test each program revision, he wants scripts to inject sample data into user apps and then another (big) program looks at EVERYTHING that comes out, comparing it to a baseline. To me, the scope of this project is huge, perhaps bigger that our ERP system. So far, I have been unable to talk him out of it. Thus, I am wondering what other people are doing in the way of software test. Virtually all of the source code that we have is in UniBasic. I do not know what jargon to use to describe the test that I do now. On a spreadsheet, I layout the various possiblities and then try each scenario... One-by-one... Fixing bugs as I find them and then starting the batch of tests from scratch. --Bill ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users Ken Hall www.old-scholls.com mailto:k...@old-scholls.com phone: 503-702-7841 ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users -- Ben Souther Manager, Web and Web Service Development bsout...@fwdco.com | 508.927.8147 FWDavison Company, Inc. 10 Cordage Park Circle, Suite 200 Plymouth, MA 02360-7318 CONFIDENTIALITY NOTICE: This e-mail message, and any accompanying documents, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure, distribution or copying is prohibited. If you are not the intended recipient, please contact our office by e-mail or by telephone at (508) 747-7261 and immediately destroy all copies of the original message. ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Software Test
And who is supposed to do your other job in the meantime? Anf who is supposed to test this thing in a couple of years time? I would tell him to shovel it and find myself a real job. Brutzman, Bill wrote: Ken: Thanks for writing. I neglected to mention that there is only one programmer here... me. --Bill -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ken Hall Sent: Tuesday, September 22, 2009 4:21 PM To: U2 Users List Subject: Re: [U2] Software Test Bill - Years ago, while working at ADP Dealer Services, we were asked to attempt the same thing. Your take on the scope of the project is correct. It is a hugh project that requires some way to maintain the scripts in sync with the code as well as to store the sets of responses to enter. A top level team of ADP developers looked into the project and concluded that it was not cost effective. Due to the ever changing nature of the tests that are required for new apps and feature changes, keeping the scripts in sync with the next version of changes requires as much manpower as the manual testing. A better strategy for serious testing is to develop a separate staff that knows the application well and just is responsible for testing and development of test plans (an art in of itself that programmers are not very good at). I believe that this would offer a more cost effective strategy and get your company better testing that unit tests done by the programmers. In some programming environments (other than MV), there are products to do this kind of instrumented testing, but the only ones I ever investigated cost more that our entire budget for the QC staff that did our testing and a lot more at ADP. Ken At 12:57 PM 9/22/2009, you wrote: My boss has a new grand scheme for software testing. To test each program revision, he wants scripts to inject sample data into user apps and then another (big) program looks at EVERYTHING that comes out, comparing it to a baseline. To me, the scope of this project is huge, perhaps bigger that our ERP system. So far, I have been unable to talk him out of it. Thus, I am wondering what other people are doing in the way of software test. Virtually all of the source code that we have is in UniBasic. I do not know what jargon to use to describe the test that I do now. On a spreadsheet, I layout the various possiblities and then try each scenario... One-by-one... Fixing bugs as I find them and then starting the batch of tests from scratch. --Bill ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users Ken Hall www.old-scholls.com mailto:k...@old-scholls.com phone: 503-702-7841 ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] Software Test
Bill: One strategy that improves testing results is to correctly define what a successful test is. A successful test is one that *finds* errors in the code. If no errors are found, then either there aren't any (and that's highly unlikely,) or the tests aren't good enough to trigger the problems. Ideally, you get a hold of someone who knows the application very well, and has a demonic streak. Have that person follow the strategy Ken Hall suggested in the earlier post. If they are immediately successful (i.e. finding errors,) track the rate of error discovery. When the rate falls below, say one per day, you might be at a cutoff point. Get a second opinion from another destructive expert if you can, particularly if the first fails (i.e. finds no errors.) Basically, nothing beats an application expert bent on breaking the software for finding bugs the fastest. What your boss is proposing to do is to kill several hundred bacteria by boiling the ocean. Managing test case coverage for a typical application is a huge task. And when you'll get a positive ROI is unclear: At what point will the test system have enough critical mass (test coverage) to actually demonstrably improve reliability and user satisfaction? Lee On Tue, 22 Sep 2009, Brutzman, Bill wrote: My boss has a new grand scheme for software testing. To test each program revision, he wants scripts to inject sample data into user apps and then another (big) program looks at EVERYTHING that comes out, comparing it to a baseline. To me, the scope of this project is huge, perhaps bigger that our ERP system. So far, I have been unable to talk him out of it. Thus, I am wondering what other people are doing in the way of software test. Virtually all of the source code that we have is in UniBasic. I do not know what jargon to use to describe the test that I do now. On a spreadsheet, I layout the various possiblities and then try each scenario... One-by-one... Fixing bugs as I find them and then starting the batch of tests from scratch. --Bill ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users -- Lee J. Leitner, Ph.D. leitn...@acm.org http://www.leitner.org/~leitnerl The world can only be grasped by action, not by contemplation. The hand is the cutting edge of the mind. -- Jacob Bronowski V.13.0 --- ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
