OK. Though Chris is but one QE engineer in the world, his arguments are largely bullet-proof. I will start looking at how to pop open this can of worms with a Pythonic access layer of some kind. Thanks too, to Stephan Diehl, for his XML preprocessor and for the pointer to SLIP. I will check out both.
I have vacation the week after next. That will be a good time to tackle this (much to my wife's chagrin). I will likely post some lead-in questions in this forum between now and then. Keyton > -----Original Message----- > From: Chris Prinos [mailto:[EMAIL PROTECTED]] > Sent: Friday, June 07, 2002 12:14 AM > To: 'A. Keyton Weissinger'; [EMAIL PROTECTED] > Subject: RE: [Webware-discuss] RE: Web testing > > > Keyton, > > Here are a few thoughts from a QA engineer that has been heavily > invested in > test automation of one kind of another for over a decade (my first qa job > was for compiler front-ends in a dev tools company, so that > helped steer my > course). > > > > CASE AGAINST SCRIPT: > > 1) My end goal is a GUI-based tool (potentially commercial > > grade or better) > > that is as easy to use as the commercial versions out there > > today (Mercury > > Astra QuickTest, Empirix's e-Test, others). The target > > audience for this > > goal is (a) quality assurance engineers; (b) developers. > > 2) I don't want you to *have* to write XML *OR* script. > > This is a nice idea to keep the barrier of entry low. And it makes for a > better 'tool' demo. This is crowded product space though, and > there are some > very mature tools (Compuware, Segue, WinRunner, etc) that make > this promise > but cannot truly provide a solution that allows adequate testing without > scripting. I don't think it's because they are bad products, but because > non-coding mechanisms for creating functional tests don't often scale very > well past simple problems. > > > 1) Until such time as I have a GUI, all users will likely be > > developers (as > > most QA folks don't have time to write code to do tests -- at > > least like > > this). > > I'd be careful about making this assumption. Some QA teams do not have > people with this skills for writing code, but it's not necessarily a time > issue. Typical automation efforts (record & play style, or > scripting) take a > significant amount of time to create and maintain. The trick is to make it > possible for QA development to keep up with engineering development. The > best way to do that is to use high level tools (like what you're > describing > for Puffin) to allow QA RAD, but not to the exclusion of code-based > solutions. I don't have to convince fellow Python fans of the benefits of > high level languages, and this is what the 'big' commercial tools provide. > The shortcomings of those environments are typically: > 1) script language is too naive > 2) script language is proprietary > 3) lack of good development/debugging tools and third party support > (because of #2) > 4) pool of people familiar with the language is small (even > with the most > popular products) > 5) difficult to integrate with other systems interfaces > 6) poor test framework/base libraries. > Not all commercial tools suffer from all the problems. WinRunner > is popular > for example, and it's easy to come up to speed on the scripting language. > But you can't truly create a test architecture using WinRunner > script (e.g., > no user defined data types). It's not object oriented. There's no test > framework to guide organization of code. You can make it do the > job, but the > results are not often pretty. Segue's 4Test (in SilkTest) is an > example of a > really nice language for development. It's dynamic, object oriented, has > facilities for built in documentation, allows indentation based coding or > c-style with braces (you can switch at any time), and has an excellent > framework for regression tests, setting up application base states, etc. > Even the first betas of that product years ago, it was very > innovative. But > if you ask Segue for the sales pitch... they'll give you the Record & > Playback Dog-and-Pony show and downlplay the fact that there's any > programming at all. But the maturity of the language let me create a test > architecture for a product that got ported from Mac OS to NT, and > the tests > were ported long before the app was, just waiting for the new app to test, > with a high degree of re-use. As good as it is though, it still creates > stumbling blocks. Need to parse XML? What are you going to do, write your > own parser in 4Test? Have a coding question? See how many > responses you get > on comp.software.testing (hint: don't hold your breath). > > I think QA test automation is actually really a potential sweet spot for > Python because, from a development standpoint, it provides all > the language > features to create well structured test code. Combining it with a proper > test framework (this is where Puffin, webunit, pyunit, etc come in) is a > powerful combination. All the reasons that make Python great for software > development hold true for test development. Few of it's disadvantageous > (speed) matter when it comes to test code. And the resources available are > tremendous. > > Here's the other thing, getting an advantage in test automation relies on > leverage. Can you leverage code from one set of test to another? > Can you use > the same tool to write a test as you do to create a test harness or > simulator (not the same thing!). > > My current forray into Webware is a good example. My app is a web > application that is being used to act as a model (simulation) for > web sites > that will use our product's web services. The real server system is > Java/J2EE (so are most of the real clients), but the model is > Python. It can > interact with the real application like it's supposed to, validate > interactions, do bad things on purpose, etc. Not only that, but the actual > tests that run are written in Python too, and they can share the same code > as the model. > > The real product uses other web-services for back end tasks. We > can't alway > use those (real systems)for testing, so we need to simulate them too. > Python. Scripts to act like a user surfing with Internet > Explorer? IE's COM > interface with Python win32com. Decrypt data from the database using our > application's proprietary Java code? Jython. Need to automate > aspects of the > source control system? -G in Perforce spits out pickeled Python > dictionaries. Can the software engineers see Jython as helpful for their > Java development? Yeah?...Now you've got another resource who will be able > answer questions with Python development for newbie QA programmers. > > > Perl enthusiast suggest the same thing from the Perl side of > > the great wall. > > Heck, I've even had some VB-slaves grunt similar comments. > > > Perl is good glue too. I'm not sure it scales well though, and I > think it's > a bigger barrier of entry for QA engineers that don't spend eight hours a > day writing code. VB based testing solutions can be pretty good > too...except you kill yourself if you need to stray off Win32. > TestMaker is > going the python (actually Jython) route, but they're making some odd > choices too (test scripts are python files, but they aren't .py, trying to > create their own IDE with NetBeans, but where's the debugger, etc). > > > So it comes down to this: How can I best (and most quickly) > > meet the needs > > of my number 1 audience (QA engineers) while continuing to > > help (and receive > > AWESOME feedback like this) from developers? > GUI based test design is a good start, but don't give up the > flexibilty that > a programming interface would provide. If your product is written > in Python, > you've got the best scripting language just sitting there, let the user's > exploit it. > > Would you > > still want a script-based version as much if a GUI prevented > > your exposure > > to the XML nastiness? In your experience with QA > > professionals, do you think > > it would be useful to QA folks? > > yes. yes. :) > > > Chris > > > _______________________________________________________________ Don't miss the 2002 Sprint PCS Application Developer's Conference August 25-28 in Las Vegas -- http://devcon.sprintpcs.com/adp/index.cfm _______________________________________________ Webware-discuss mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/webware-discuss
