I am having trouble getting my messages posted to the list. This one
is taking a long time and I wonder if it is because it has HTML in the
content and an attachment. I am pasting it in plain test.
...
One of the presentations at the Users Conference will be me talking on
my experiences with Grinder. With David there, we hope to go far past
that to discuss strategies for adding testing functionality to OFBiz.
I have no experience that qualifies me to be an expert in this area -
just have a need. Rather than just wait for everyone to show up at the
conference, I felt it would be helpful to try to treat some of the
subjects ahead of time. Of course, not everything will be decided at
the conference, so as much input as possible from everyone will be
needed. Also, if there is anyone with strong Python and/or Grinder
skills, their help would also be appreciated.
OFBiz Testing Initiative
GOAL
BACKGROUND
Problem with Design Documents
Applicable Features of OFBiz
Applicable Features of Grinder
Applicable Features of Extreme Programming
REQUIREMENTS
Easy Generation
Handle Complex Tests
Adaptable per Application
Rugged
Handle Iterations
Usable at Multiple Proficiency Levels
Simplify Output Testing
STRAWMAN PLAN
Modify Test Script Generation
Add Test Overlay Config File
Source Code Control to Handle Iterations
Appendices
Standard Generated Grinder Test Script
Possible Custom Generated Grinder Test Script
Possible Overlay XML File
GOAL The goal of the OFBiz Testing Initiative (OFBTI) is to add
functionality to OFBiz that will streamline the functional testing
process to the point that it will be cost-effective and beneficial to
write comprehensive tests. Ideally, the tools could be used by
non-programmers.
A secondary goal would be to add tools to make design specs easier to
write and more useful.
BACKGROUND
Problem with Design Documents Design documents, though necessary, have
serious drawbacks.
Expensive to create - should probably take as much time as the
implementation.
Never complete enough - always need to go back to client, anyway.
No automation help in implementation and validation - as text
documents, they must be manually translated into architecture
documents.
Easy to get out-of-date - while many times they are supposed to be
"living" documents, in reality, they almost never are.
Applicable Features of Extreme Programming Extreme programming
addresses many of the shortcomings of the more conventional design
document approach. Instead of a complete design doc or even use case
up front, all that is required is a "story". The story needs only be
complete enough to generate the first test and the first test is
usually just a placeholder. Then in the course of many iterations, the
functional test is enhanced, not the design document. The tests are
living and serve the useful purpose of verifying that things have not
broken as code gets changed.
One of the very useful things about this approach is that when bugs
are discovered, a test or subtest is added to guard against it ever
creeping back into the code.
Applicable Features of OFBiz OFBiz is different from other development
environments, and it would be good to identify the features that could
be used advantageously or which must be dealt with.
OFBiz is highly configurable via XML formatted files and it would be
good to continue that pattern. XML config files, because they have
associated schema, make it feasible for non-programmers and those not
intimately familiar with OFBiz to make changes and perform certain
programming tasks.
OFBiz has a general pattern of offering easily configurable options
via data parameters, but always offering easy access to the lowest
level of programming for those needed cases without requiring a huge
environment adaptation. For instance, the screen widget config files
allow screens to be created with miminal data and allow the user to be
prompted via the schema in an XML editor, but if there is a construct
that the widgets do not handle, the user can just throw in a call to
an HTML component or a FreeMarker template - they do not need to
abandon the screen widget framework.
OFBiz is a service oriented framework and there are many tools that
can be used aid in that process. If a request is processed by a
service (in lieu of an event) then the input parameters can be
automatically taken from the HTTP stream and automatically converted
to the right type. Also, the form widget can build forms from the
service definition.
Applicable Features of Grinder There are multiple options for testing
frameworks, but Grinder offers the following unique advantages.
A robust test script generator - TCPProxy attempts to assign return
parameters to variables and reuse them, rather than passing
literals around. This means that it does not instantly break when keys
are autogenerated.
Grinder has convenient customization point - script generation is done
via an XSLT stylesheet that makes use of Java extensions. This would
be a natural place to make modifications to suit OFBTI. There are also
filters for the requests and responses that can be swapped out via
command line parameters.
Jython - allows seamless switching between a highly productive
scripting environment and regular Java code. Because of this feature,
Jython has a lot of possibilities within OFBiz.
Integrated functional and jUnit testing - the use of Jython allows for
the same environment to be used to run HTTP client system tests and
jUnit functional tests.
REQUIREMENTS
Easy Generation The use of a test script generator, such as Grinder's
TCPProxy, allows test scripts to be created by capturing the input of
a user at a browser.
Handle Complex Tests A further enhancement would be to allow the
creation of scripts by supplying a few data parameters or by easily
modifying automatically generated scripts. One of the biggest
drawbacks of end user testing environments is that when something
changes, the associated test script is usually unusable. A big
improvement would be to allow tests to be created by chaining together
subtests.
Adaptable per Application Many applications will have special
characteristics that need to be handled specially. In one instance, I
found that Grinder was using the '$' character to form Python variable
names (which doesn't work) because that is how the HTTP parameters
were named.
Rugged
Handle Iterations The general idea of XP is that the tests become more
complex and comprehensive as the code becomes more complete. It would
probably be a good idea to retain visibility to past test setups -
though, at this time, I am not sure why.
Usable at Multiple Proficiency Levels It would be good to have the
testing environment useable, or at least understandable, by
non-programmers. JUnit tests would not meet this requirement. Neither
would Python scripts (though it could come close). For project
managers, an XML-based configuration environment would be needed.
Simplify Output Testing The analysis of the HTML pages that are
returned by Grinder tests is one of the more problematic areas of
end-user testing. The more common scenarios would need to be handled
by the XML-based configuration environment mentioned above. The
foreseeable cases would be matching of form values, existence or
non-existence anywhere of a phrase or something that can be tested by
an XPath expression.
STRAWMAN PLAN What is discussed below is how I see tackling the
testing problem. I am only interested in what will be supported by the
community, so it can be changed.
The overall approach is to use as much of Grinder as possible, and
allow for the generated Grinder test scripts to be overwritten by a
user generated XML test configuration file.
Modify Test Script Generation The Grinder TCPProxy program will be
used, but the XSLT stylesheet and the associated Java extension
classes will be enhanced. The main change will be that the individual
request tests that Grinder generates for each round trip to the server
will be wrapped by code that will check to see if there are data
overrides to be made coming from the user generated high level test
config file. Each wrapper will also have a dictionary (ie. map) that
defines tests to be made on each low level result. These also will be
modified by the higher level config file. If there is not higher level
config file, the script as generated by TCPProxy will run with no
modifications.
Input Dictionary The input directory will do more than just allow the
user to supply literal values as test input. There will be helper
functions for allowing the user to randomly pick values from a list.
Also there will be helper functions for generating reasonably looking
addresses. The user will be able to create Python scripts to generated
special input. See the appendix for samples of what this would look
like.
Output Dictionary The output dictionary, which contains the test
criteria, would have test helper functions of the following sort:
Literal value
Regular expression
Form value
XPath
Custom Jython scripts Many of the test methods will be suitable for
non-programmers, but the use of Jython scripts follows the OFBiz
pattern of providing simple methods for simple tasks, but making it
easy to drop down to the level needed to solve the problem. At some
point (right away?), it will be necessary to allow complex joing of
tests with AND, OR and NOT operators.
Add Test Overlay Config File One of the modifications to the OOTB test
script generated by TCPProxy will be that the script will look for the
existence a file path as a parameters and use it to overwrite the
default test script values (ie. the values typed in by the user when
the script was being generated). The script could have multiple levels
of complex scripts, but eventually, they must call one of the
"request" tests. Keep in mind that the base test script could have a
large number of "request" scripts, but they would not all need to be
used by the user-defined config file; it could just use a subset. So
the base test script may not, in fact, be totally generated by one
session of user interaction with the system under test; it could be
built up over time as new functionality is added without having to run
thru the test. The OOTB behavior is to sequentially number the tests,
but in this case, we may wish to name them with the request name that
they interact with.
These high level test scripts would need "include" functionality so
that they could used standard building blocks.
Source Code Control to Handle Iterations
Appendices
Standard Generated Grinder Test Script This is the actual output from
a TCPProxy session:
# The Grinder 3.0-beta30
# HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM
from net.grinder.script import Test
from net.grinder.script.Grinder import grinder
from net.grinder.plugin.http
import HTTPPluginControl, HTTPRequest
from HTTPClient import NVPair
connectionDefaults = HTTPPluginControl.getConnectionDefaults()
httpUtilities = HTTPPluginControl.getHTTPUtilities()
# To use a proxy server, uncomment the next line and set the host
and port.
# connectionDefaults.setProxyServer("localhost", 8001)
# These definitions at the top level of the file are evaluated once,
# when the worker process is started.
connectionDefaults.defaultHeaders
= \
( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'),
NVPair('Accept-Encoding', 'gzip,deflate'),
NVPair('Accept-Language', 'en-us,en;q=
0.5'),
NVPair('Accept-Charset', 'UTF-8,*'),
NVPair('Accept',
'text/xml,application/xml,application/xhtml+xml,text/html;q=
0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), )
headers0= \
( )
headers1= \
( NVPair('Referer', 'https://localhost:8443/webtools/control/
main' ), )
headers2= \
( NVPair('Referer', '
https://localhost:8443/webtools/control/checkLogin/main'), )
url0 = ' https://localhost:8443'
# Create an HTTPRequest for each request, then replace the
# reference to the HTTPRequest with an instrumented version.
# You can access the unadorned instance using request101.__target__.
request101 = HTTPRequest(url=url0, headers=headers0)
request101 = Test(101, 'GET /').wrap(request101)
request102 = HTTPRequest(url=url0, headers=headers0)
request102 = Test(102, 'GET main').wrap(request102)
request201 = HTTPRequest(url=url0, headers=headers1)
request201 = Test(201, 'GET main').wrap(request201)
request301 = HTTPRequest(url=url0, headers=headers2)
request301 = Test(301, 'POST login').wrap(request301)
class TestRunner:
"""A TestRunner instance is created for each worker thread."""
# A method for each recorded page.
def page1(self):
"""GET / (requests 101-102)."""
# Expecting 302 'Moved Temporarily'
result = request101.GET('/webtools/')
grinder.sleep(16)
request102.GET('/webtools/control/main')
return result
def page2(self):
"""GET main (request 201)."""
result = request201.GET('/webtools/control/checkLogin/main')
return result
def page3(self):
"""POST login (request 301)."""
result = request301.POST('/webtools/control/login',
( NVPair('USERNAME', 'admin'),
NVPair('PASSWORD', 'ofbiz'), ),
( NVPair('Content-Type', 'application/x-www-form-urlencoded'), ))
return result
def __call__(self):
"""This method is called for every run performed by the worker
thread."""
self.page1() # GET / (requests 101-102)
grinder.sleep(4328)
self.page2() # GET main (request 201)
grinder.sleep(2109)
self.page3() # POST login (request 301)
def instrumentMethod(test, method_name, c=TestRunner):
"""Instrument a method with the given Test."""
unadorned = getattr(c, method_name)
import new
method = new.instancemethod
(test.wrap(unadorned), None, c)
setattr(c, method_name, method)
# Replace each method with an instrumented version.
# You can call the unadorned method using self.page1.__target__().
instrumentMethod(Test(100, 'Page 1'), 'page1')
instrumentMethod(Test(200, 'Page 2'), 'page2')
instrumentMethod(Test(300, 'Page 3'), 'page3')
Possible Custom Generated Grinder Test Script # The Grinder 3.0-beta30
# HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM
from net.grinder.script import Test
from net.grinder.script.Grinder import grinder
from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest
from HTTPClient import NVPair
import AGrinderTest
connectionDefaults = HTTPPluginControl.getConnectionDefaults()
httpUtilities = HTTPPluginControl.getHTTPUtilities()
# To use a proxy server, uncomment the next line and set the host
and port.
# connectionDefaults.setProxyServer("localhost", 8001)
# These definitions at the top level of the file are evaluated once,
# when the worker process is started.
connectionDefaults.defaultHeaders
= \
( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1;
en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'),
NVPair('Accept-Encoding', 'gzip,deflate'),
NVPair('Accept-Language', 'en-us,en;q=
0.5'),
NVPair('Accept-Charset', 'UTF-8,*'),
NVPair('Accept',
'text/xml,application/xml,application/xhtml+xml,text/
html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'),
)
headers0= \
( )
headers1= \
( NVPair('Referer', 'https://localhost:8443/webtools/control/
main'), )
headers2= \
( NVPair('Referer', '
https://localhost:8443/webtools/control/checkLogin/main'), )
url0 = 'https://localhost:8443'
# Create an HTTPRequest for each request, then replace the
# reference to the HTTPRequest with an instrumented version.
# You can access the unadorned instance using request101.__target__.
request101 = HTTPRequest(url=url0, headers=headers0)
request101 = Test(101, 'GET /').wrap(request101)
request102 = HTTPRequest(url=url0, headers=headers0)
request102 = Test(102, 'GET main').wrap(request102)
request201 = HTTPRequest(url=url0, headers=headers1)
request201 = Test(201, 'GET main').wrap(request201)
request301 = HTTPRequest(url=url0, headers=headers2)
request301 = Test(301, 'POST login').wrap(request301)
class TestRunner:
"""A TestRunner instance is created for each worker thread."""
# A method for each recorded page.
def page1(self):
"""GET / (requests 101-102)."""
# Expecting 302 'Moved Temporarily'
result = request101.GET('/webtools/')
grinder.sleep(16)
request102.GET('/webtools/control/main')
return result
def page2(self):
"""GET main (request 201)."""
result = request201.GET('/webtools/control/checkLogin/main')
return result
def page3(self):
"""POST login (request 301)."""
result = request301.POST('/webtools/control/login',
( NVPair('USERNAME', 'admin'),
NVPair('PASSWORD', 'ofbiz'), ),
( NVPair('Content-Type', 'application/x-www-form-urlencoded'), ))
return result
"""This is not working code. Just an idea of how the generated
code would look"""
def webtoolsWrap(self, overlayMap):
inMap = {}
outMap = {}
self.agrinder.runWrappedTest(page1, inMap, outMap, overlayMap)
def mainWrap(self, overlayMap):
inMap = {}
outMap = {}
self.agrinder.runWrappedTest(page2, inMap, outMap, overlayMap)
def loginWrap(self, overlayMap):
inMap = {'USERNAME':'admin', 'PASSWORD':'ofbiz'}
outMap = {}
self.agrinder.runWrappedTest
(page2, inMap, outMap, overlayMap)
def __call__(self):
"""This method is called for every run performed by the worker
thread."""
"""I am going to do some handwaving here because I want to get
this doc out today,
but there would be code here to read in an argument from the
command line and
use it as a file path to read in the "test overlay" XML doc"""
self.agrinder = AGrinderTest.AGrinder
()
testNode = self.agrinder.getOverlayTest("webtools", sys.argv)
self.webtoolsWrap(testNode) # GET / (requests 101-102)
grinder.sleep(4328)
self.mainWrap(testNode) # GET main (request 201)
grinder.sleep(2109)
self.loginWrap(testNode) # POST login (request 301)
def instrumentMethod(test, method_name, c=TestRunner):
"""Instrument a method with the given Test."""
unadorned = getattr(c, method_name)
import new
method = new.instancemethod(test.wrap(unadorned), None, c)
setattr(c, method_name, method)
# Replace each method with an instrumented version.
# You can call the unadorned method using self.page1.__target__().
instrumentMethod(Test(100, 'Page 1'), 'page1')
instrumentMethod(Test(200, 'Page 2'), 'page2')
instrumentMethod(Test(300, 'Page 3'), 'page3')
Possible Overlay XML File This example is probably not even close to
what the final test overlay XML script will look like, but I think it
is better to have something to go from. <tests>
<test name="webtools">
<subtest name="login">
<input>
<field name="USERNAME">jdoe</field>
<field name="PASSWORD">id9Ed3jk</field>
</input>
<match>
<field type="regex" op="not">not found</field>
<field type="script" op="true">isSuccess()</field>
</match>
</subtest>
</test>
</tests>
