Greetings. Allow me to spend a few moments talking about Puffin.
Puffin is an extensible web application regression testing framework. This
is slightly different than the WebUnit, HttpUnit and other projects. Puffin
is aimed at functionally testing an entire site. These others are aimed more
at unit testing pages. While Puffin can do this, it is not its main purpose.
That said, I firmly believe there is room (and desperate need) for both the
WebUnits of the world and the Puffins of the world. I freely admit I
shamelessly stole some great ideas from Stephen ;-).
In a nutshell, with Puffin, you have a configuration file that has two
sections. The first tells puffin stuff like what server you wish to use, how
you want it to tell you about its results (via command line, file, etc) and
a few other system type things. Here is a sampling:
<puffin>
<!-- System configuration settings. -->
<system>
<server host="localhost" port="80"/>
<frameworkLogging channelPriority="WARN">
<handler type="StreamHandler">
<param name="msgFormat"><![CDATA[%(message)s]]></param>
</handler>
</frameworkLogging>
<reportLogging channelPriority="DEBUG">
<handler type="FileHandler">
<param name="msgFormat" eval="0"><![CDATA[%(message)s]]></param>
<param name="fileName" eval="0">puffinResults.log</param>
<param name="mode" eval="0"><![CDATA[a+]]></param>
</handler>
</reportLogging>
<failureAlertLogging channelPriority="DEBUG">
<handler type="FileHandler">
<param name="msgFormat" eval="0"><![CDATA[%(message)s]]></param>
<param name="fileName" eval="0">puffinFailures.log</param>
<param name="mode" eval="0"><![CDATA[a+]]></param>
</handler>
</failureAlertLogging>
<resultsProcessor reportDetail="ALL"/>
<defaultResponseAnalyzerList>
<responseAnalyzer src="puffin" type="TimerResponseAnalyzer">
<param name="timeLimitSecs">10</param>
</responseAnalyzer>
<responseAnalyzer src="puffin" type="StatusResponseAnalyzer">
<param name="httpStatus">200</param>
<param name="httpStatus">302</param>
</responseAnalyzer>
</defaultResponseAnalyzerList>
</system>
.
.
.
The second part of the config file tells Puffin about the various "test
actions" that are possible. A test action is the smallest atomic action you
will take as part of your test. These can be brought together in a myriad
different ways to create test plans (see below) of endless complexity. Here
are a few test action configurations:
<testAction name='login'>
<path>/puffindemoapp/start.cgi</path>
<!-- INPUTS -->
<!-- OUTPUTS -->
<output name="SESSION_ID" processor="extractRegex">
<param name="expr"
eval="0"><![CDATA[href="itemList\.cgi\?session_id=(\d*)]]></param>
</output>
</testAction>
<testAction name='getPetsList'>
<path>/puffindemoapp/itemList.cgi</path>
<input name="list" type="GET" processor="VALUE">
<param name='value'>Pets</param>
</input>
<input name="session_id" type="GET" processor="DICT">
<param name='key'>SESSION_ID</param>
</input>
</testAction>
<testAction name="selectRandomCategory" noSvrCall="1">
<output name="SELECTED_CATEGORY" processor="LISTACCESS">
<param name="listToken">ITEM_CATEGORY_LIST</param>
<param name="index">*</param>
</output>
</testAction>
As you can see, a test action can involve a call to the server or it can
not.
You test the "validity" of a test action's result using a thing called a
ResponseAnalyzer which can be configured for all test actions (as in the
case above) or specific test actions.
So you might want all test actions that involve an HTTP call to be checked
for a correct HTTP status, but you may want to test a specific test action
for the existence of the phrase "EXCEPTION OCCURRED. PLEASE ENTER A NAME."
or similar. Puffin allows for either (or both) scenario(s).
Finally, in addition to a configuration plan you can create one or more test
plans. Puffin allows for two formats of test plans. The first is a simple
format which just lists test action names, like this:
# Comments start with a "#"
login
getPetsList
selectRandomCategory
Or you can use a far more flexible complex format that allows for
dependencies, repetition, and transactionality (by wrapping multiple test
actions together in a unit that pass or fail as a whole):
<plan>
<task name="login">
<testAction name="resetInventory"/>
<testAction name="login"/>
</task>
<subPlan>testPlans/subPlan_dW2.xml</subPlan>
<task name="makePurchase" depends="makeDecision" repeat="2">
<testAction name="generateSelectionCount"/>
<testAction name="addItemToCart"/>
<testAction name="updateCurrentSelection"/>
<testAction name="showCart"/>
<testAction name="getUpdatedInventory"/>
</task>
<task name="checkOut" depends="makePurchase">
<testAction name="checkOut"/>
</task>
</plan>
As you can see, you can include plans within plans within plans (as Leto
Atreides would say). (You can do this with the configuration file too --
include files containing other test actions.)
I'm in the process of getting a big release out in the next couple days
which will make stuff like testing forms SIGNIFICANTLY easier (it can get
WAY too lengthy right now) and will allow more flexibility in what you can
repeat in a test plan. Here's an example of the next generation complex test
plan:
<plan>
<repeat until="*">
<task name="login">
<testAction name="resetInventory"/>
<testAction name="login"/>
</task>
<repeat count="4">
<task name="makeDecision" depends="login">
<testAction name="selectRandomCategory"/>
<testAction name="getItemLists"/>
<testAction name="selectItem"/>
</task>
<repeat while="CURRENT_INVENTORY_COUNT">
<task name="makePurchase" depends="makeDecision">
<testAction name="generateSelectionCount"/>
<testAction name="addItemToCart"/>
<testAction name="updateCurrentSelection"/>
<testAction name="showCart"/>
<testAction name="getUpdatedInventory"/>
</task>
</repeat>
</repeat>
<task name="checkOut" depends="makePurchase"/>
</repeat>
</plan>
While this may at first offend your XML sensibilities (it continues to
offend mine), it DOES allow for an AMAZING amount of test flexibility.
Extensibility: You can create your own response analyzers, output
processors, etc using Python, etc. and Puffin will use them as easily as it
does its own.
Other goodies: MySQL support. Support for command line calls. Rich regular
expression support. Ability to test web services and other XML-based
responses, etc. XPath-based output extraction.
The down side:
Puffin XML can get WAY too long. Make sure you use an XML Spy or similar
product to edit the test action configuration which can get very verbose
with many different possibilities. I'm addressing this to some extent in the
next release, but my long term goal is to build a GUI on top of it (ala
Empirix's e-Test, etc) to hide this pain. If I can get the quality where I
want it, I may make a go of a commercial version of the GUI, but some
version of it will -- of course -- stay open source (as will the Puffin core
itself).
In the interest of full disclosure, Ken MacLeod of your number worked with
Puffin for a while and put it down because of this complexity. He was doing
some pretty lengthy form validation and the test action configuration XML
just got too nasty. I hope that he gives it another shot in the next release
but publicly THANK him for giving me his rich and very constructive
feedback, regardless -- THANK YOU KEN! A huge part of the next release is
being built in direct response to his feedback.
(A colleage -- Bryan Richard, editor of Py -- and I are also working on a
proxy-based recorder to help with test action configuration and simple test
plan creation. Look for the first alpha of this in the coming weeks.)
Also, creating test actions in such a way as to allow for flexible test
plans is a pretty subtle exercise, which takes some thought. I'm addressing
this by covering this topic in a series of articles for IBM's developerWorks
(the first of which should hit the site "real soon" according to its
editors).
Give it a shot. I am in the midst of a biggie refactor in response to Ken's
feedback and to add stuff like POP3 test action support and the like. I want
the thing to be commercial-grade and better so I try very diligently to
respond quickly to user requests.
Take care, everyone!
Keyton
> -----Original Message-----
> From: Chuck Esterbrook [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, June 06, 2002 3:19 AM
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> Subject: Web testing
>
>
> Has anyone ever used or investigated both of these? e.g., how do they
> compare?
>
> http://puffin.sourceforge.net/
> http://webunit.sourceforge.net/
>
>
> And I suppose the authors can throw in their comments, if they like!
>
> -Chuck
>
>
_______________________________________________________________
Don't miss the 2002 Sprint PCS Application Developer's Conference
August 25-28 in Las Vegas -- http://devcon.sprintpcs.com/adp/index.cfm
_______________________________________________
Webware-discuss mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/webware-discuss
