Keyton,
Here are a few thoughts from a QA engineer that has been heavily invested in
test automation of one kind of another for over a decade (my first qa job
was for compiler front-ends in a dev tools company, so that helped steer my
course).
>
> CASE AGAINST SCRIPT:
> 1) My end goal is a GUI-based tool (potentially commercial
> grade or better)
> that is as easy to use as the commercial versions out there
> today (Mercury
> Astra QuickTest, Empirix's e-Test, others). The target
> audience for this
> goal is (a) quality assurance engineers; (b) developers.
> 2) I don't want you to *have* to write XML *OR* script.
This is a nice idea to keep the barrier of entry low. And it makes for a
better 'tool' demo. This is crowded product space though, and there are some
very mature tools (Compuware, Segue, WinRunner, etc) that make this promise
but cannot truly provide a solution that allows adequate testing without
scripting. I don't think it's because they are bad products, but because
non-coding mechanisms for creating functional tests don't often scale very
well past simple problems.
> 1) Until such time as I have a GUI, all users will likely be
> developers (as
> most QA folks don't have time to write code to do tests -- at
> least like
> this).
I'd be careful about making this assumption. Some QA teams do not have
people with this skills for writing code, but it's not necessarily a time
issue. Typical automation efforts (record & play style, or scripting) take a
significant amount of time to create and maintain. The trick is to make it
possible for QA development to keep up with engineering development. The
best way to do that is to use high level tools (like what you're describing
for Puffin) to allow QA RAD, but not to the exclusion of code-based
solutions. I don't have to convince fellow Python fans of the benefits of
high level languages, and this is what the 'big' commercial tools provide.
The shortcomings of those environments are typically:
1) script language is too naive
2) script language is proprietary
3) lack of good development/debugging tools and third party support
(because of #2)
4) pool of people familiar with the language is small (even with the most
popular products)
5) difficult to integrate with other systems interfaces
6) poor test framework/base libraries.
Not all commercial tools suffer from all the problems. WinRunner is popular
for example, and it's easy to come up to speed on the scripting language.
But you can't truly create a test architecture using WinRunner script (e.g.,
no user defined data types). It's not object oriented. There's no test
framework to guide organization of code. You can make it do the job, but the
results are not often pretty. Segue's 4Test (in SilkTest) is an example of a
really nice language for development. It's dynamic, object oriented, has
facilities for built in documentation, allows indentation based coding or
c-style with braces (you can switch at any time), and has an excellent
framework for regression tests, setting up application base states, etc.
Even the first betas of that product years ago, it was very innovative. But
if you ask Segue for the sales pitch... they'll give you the Record &
Playback Dog-and-Pony show and downlplay the fact that there's any
programming at all. But the maturity of the language let me create a test
architecture for a product that got ported from Mac OS to NT, and the tests
were ported long before the app was, just waiting for the new app to test,
with a high degree of re-use. As good as it is though, it still creates
stumbling blocks. Need to parse XML? What are you going to do, write your
own parser in 4Test? Have a coding question? See how many responses you get
on comp.software.testing (hint: don't hold your breath).
I think QA test automation is actually really a potential sweet spot for
Python because, from a development standpoint, it provides all the language
features to create well structured test code. Combining it with a proper
test framework (this is where Puffin, webunit, pyunit, etc come in) is a
powerful combination. All the reasons that make Python great for software
development hold true for test development. Few of it's disadvantageous
(speed) matter when it comes to test code. And the resources available are
tremendous.
Here's the other thing, getting an advantage in test automation relies on
leverage. Can you leverage code from one set of test to another? Can you use
the same tool to write a test as you do to create a test harness or
simulator (not the same thing!).
My current forray into Webware is a good example. My app is a web
application that is being used to act as a model (simulation) for web sites
that will use our product's web services. The real server system is
Java/J2EE (so are most of the real clients), but the model is Python. It can
interact with the real application like it's supposed to, validate
interactions, do bad things on purpose, etc. Not only that, but the actual
tests that run are written in Python too, and they can share the same code
as the model.
The real product uses other web-services for back end tasks. We can't alway
use those (real systems)for testing, so we need to simulate them too.
Python. Scripts to act like a user surfing with Internet Explorer? IE's COM
interface with Python win32com. Decrypt data from the database using our
application's proprietary Java code? Jython. Need to automate aspects of the
source control system? -G in Perforce spits out pickeled Python
dictionaries. Can the software engineers see Jython as helpful for their
Java development? Yeah?...Now you've got another resource who will be able
answer questions with Python development for newbie QA programmers.
> Perl enthusiast suggest the same thing from the Perl side of
> the great wall.
> Heck, I've even had some VB-slaves grunt similar comments.
>
Perl is good glue too. I'm not sure it scales well though, and I think it's
a bigger barrier of entry for QA engineers that don't spend eight hours a
day writing code. VB based testing solutions can be pretty good
too...except you kill yourself if you need to stray off Win32. TestMaker is
going the python (actually Jython) route, but they're making some odd
choices too (test scripts are python files, but they aren't .py, trying to
create their own IDE with NetBeans, but where's the debugger, etc).
> So it comes down to this: How can I best (and most quickly)
> meet the needs
> of my number 1 audience (QA engineers) while continuing to
> help (and receive
> AWESOME feedback like this) from developers?
GUI based test design is a good start, but don't give up the flexibilty that
a programming interface would provide. If your product is written in Python,
you've got the best scripting language just sitting there, let the user's
exploit it.
Would you
> still want a script-based version as much if a GUI prevented
> your exposure
> to the XML nastiness? In your experience with QA
> professionals, do you think
> it would be useful to QA folks?
yes. yes. :)
Chris
_______________________________________________________________
Don't miss the 2002 Sprint PCS Application Developer's Conference
August 25-28 in Las Vegas -- http://devcon.sprintpcs.com/adp/index.cfm
_______________________________________________
Webware-discuss mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/webware-discuss
