Si,
Thanks for the feedback. I debated over the xml language. I think it
is important to involve a different level of user. But the safe thing
to do would be to create the api that you mention and then wrap it in
an xml layer later on if there is demand.
-Al
On 11/2/06, Si Chen <[EMAIL PROTECTED]> wrote:
> Al,
>
> This is great that you're doing this. I think a testing framework is
> really what we need right now.
>
> My suggestion is instead of creating an xml language like you show at
> the bottom to make some API methods that could be called in Jython
> scripts. I think that might be easier for people to work with and
> would require less work over time in maintaining and extending the
> xml language.
>
> On Nov 1, 2006, at 10:49 AM, Al Byers wrote:
>
> > I am having trouble getting my messages posted to the list. This one
> > is taking a long time and I wonder if it is because it has HTML in the
> > content and an attachment. I am pasting it in plain test.
> > ...
> > One of the presentations at the Users Conference will be me talking on
> > my experiences with Grinder. With David there, we hope to go far past
> > that to discuss strategies for adding testing functionality to OFBiz.
> > I have no experience that qualifies me to be an expert in this area -
> > just have a need. Rather than just wait for everyone to show up at the
> > conference, I felt it would be helpful to try to treat some of the
> > subjects ahead of time. Of course, not everything will be decided at
> > the conference, so as much input as possible from everyone will be
> > needed. Also, if there is anyone with strong Python and/or Grinder
> > skills, their help would also be appreciated.
> >
> >
> > OFBiz Testing Initiative
> > GOAL
> > BACKGROUND
> > Problem with Design Documents
> > Applicable Features of OFBiz
> > Applicable Features of Grinder
> > Applicable Features of Extreme Programming
> > REQUIREMENTS
> > Easy Generation
> > Handle Complex Tests
> > Adaptable per Application
> > Rugged
> > Handle Iterations
> > Usable at Multiple Proficiency Levels
> > Simplify Output Testing
> > STRAWMAN PLAN
> > Modify Test Script Generation
> > Add Test Overlay Config File
> > Source Code Control to Handle Iterations
> > Appendices
> > Standard Generated Grinder Test Script
> > Possible Custom Generated Grinder Test Script
> > Possible Overlay XML File
> > GOAL The goal of the OFBiz Testing Initiative (OFBTI) is to add
> > functionality to OFBiz that will streamline the functional testing
> > process to the point that it will be cost-effective and beneficial to
> > write comprehensive tests. Ideally, the tools could be used by
> > non-programmers.
> >
> > A secondary goal would be to add tools to make design specs easier to
> > write and more useful.
> > BACKGROUND
> > Problem with Design Documents Design documents, though necessary, have
> > serious drawbacks.
> > Expensive to create - should probably take as much time as the
> > implementation.
> >
> > Never complete enough - always need to go back to client, anyway.
> >
> > No automation help in implementation and validation - as text
> > documents, they must be manually translated into architecture
> > documents.
> >
> > Easy to get out-of-date - while many times they are supposed to be
> > "living" documents, in reality, they almost never are.
> >
> >
> > Applicable Features of Extreme Programming Extreme programming
> > addresses many of the shortcomings of the more conventional design
> > document approach. Instead of a complete design doc or even use case
> > up front, all that is required is a "story". The story needs only be
> > complete enough to generate the first test and the first test is
> > usually just a placeholder. Then in the course of many iterations, the
> > functional test is enhanced, not the design document. The tests are
> > living and serve the useful purpose of verifying that things have not
> > broken as code gets changed.
> >
> > One of the very useful things about this approach is that when bugs
> > are discovered, a test or subtest is added to guard against it ever
> > creeping back into the code.
> > Applicable Features of OFBiz OFBiz is different from other development
> > environments, and it would be good to identify the features that could
> > be used advantageously or which must be dealt with.
> >
> > OFBiz is highly configurable via XML formatted files and it would be
> > good to continue that pattern. XML config files, because they have
> > associated schema, make it feasible for non-programmers and those not
> > intimately familiar with OFBiz to make changes and perform certain
> > programming tasks.
> >
> > OFBiz has a general pattern of offering easily configurable options
> > via data parameters, but always offering easy access to the lowest
> > level of programming for those needed cases without requiring a huge
> > environment adaptation. For instance, the screen widget config files
> > allow screens to be created with miminal data and allow the user to be
> > prompted via the schema in an XML editor, but if there is a construct
> > that the widgets do not handle, the user can just throw in a call to
> > an HTML component or a FreeMarker template - they do not need to
> > abandon the screen widget framework.
> >
> > OFBiz is a service oriented framework and there are many tools that
> > can be used aid in that process. If a request is processed by a
> > service (in lieu of an event) then the input parameters can be
> > automatically taken from the HTTP stream and automatically converted
> > to the right type. Also, the form widget can build forms from the
> > service definition.
> > Applicable Features of Grinder There are multiple options for testing
> > frameworks, but Grinder offers the following unique advantages.
> > A robust test script generator - TCPProxy attempts to assign return
> > parameters to variables and reuse them, rather than passing
> > literals around. This means that it does not instantly break when keys
> > are autogenerated.
> > Grinder has convenient customization point - script generation is done
> > via an XSLT stylesheet that makes use of Java extensions. This would
> > be a natural place to make modifications to suit OFBTI. There are also
> > filters for the requests and responses that can be swapped out via
> > command line parameters.
> > Jython - allows seamless switching between a highly productive
> > scripting environment and regular Java code. Because of this feature,
> > Jython has a lot of possibilities within OFBiz.
> > Integrated functional and jUnit testing - the use of Jython allows for
> > the same environment to be used to run HTTP client system tests and
> > jUnit functional tests.
> > REQUIREMENTS
> > Easy Generation The use of a test script generator, such as Grinder's
> > TCPProxy, allows test scripts to be created by capturing the input of
> > a user at a browser.
> > Handle Complex Tests A further enhancement would be to allow the
> > creation of scripts by supplying a few data parameters or by easily
> > modifying automatically generated scripts. One of the biggest
> > drawbacks of end user testing environments is that when something
> > changes, the associated test script is usually unusable. A big
> > improvement would be to allow tests to be created by chaining together
> > subtests.
> > Adaptable per Application Many applications will have special
> > characteristics that need to be handled specially. In one instance, I
> > found that Grinder was using the '$' character to form Python variable
> > names (which doesn't work) because that is how the HTTP parameters
> > were named.
> > Rugged
> > Handle Iterations The general idea of XP is that the tests become more
> > complex and comprehensive as the code becomes more complete. It would
> > probably be a good idea to retain visibility to past test setups -
> > though, at this time, I am not sure why.
> > Usable at Multiple Proficiency Levels It would be good to have the
> > testing environment useable, or at least understandable, by
> > non-programmers. JUnit tests would not meet this requirement. Neither
> > would Python scripts (though it could come close). For project
> > managers, an XML-based configuration environment would be needed.
> > Simplify Output Testing The analysis of the HTML pages that are
> > returned by Grinder tests is one of the more problematic areas of
> > end-user testing. The more common scenarios would need to be handled
> > by the XML-based configuration environment mentioned above. The
> > foreseeable cases would be matching of form values, existence or
> > non-existence anywhere of a phrase or something that can be tested by
> > an XPath expression.
> > STRAWMAN PLAN What is discussed below is how I see tackling the
> > testing problem. I am only interested in what will be supported by the
> > community, so it can be changed.
> >
> > The overall approach is to use as much of Grinder as possible, and
> > allow for the generated Grinder test scripts to be overwritten by a
> > user generated XML test configuration file.
> > Modify Test Script Generation The Grinder TCPProxy program will be
> > used, but the XSLT stylesheet and the associated Java extension
> > classes will be enhanced. The main change will be that the individual
> > request tests that Grinder generates for each round trip to the server
> > will be wrapped by code that will check to see if there are data
> > overrides to be made coming from the user generated high level test
> > config file. Each wrapper will also have a dictionary (ie. map) that
> > defines tests to be made on each low level result. These also will be
> > modified by the higher level config file. If there is not higher level
> > config file, the script as generated by TCPProxy will run with no
> > modifications.
> > Input Dictionary The input directory will do more than just allow the
> > user to supply literal values as test input. There will be helper
> > functions for allowing the user to randomly pick values from a list.
> > Also there will be helper functions for generating reasonably looking
> > addresses. The user will be able to create Python scripts to generated
> > special input. See the appendix for samples of what this would look
> > like.
> > Output Dictionary The output dictionary, which contains the test
> > criteria, would have test helper functions of the following sort:
> > Literal value
> > Regular expression
> > Form value
> > XPath
> > Custom Jython scripts Many of the test methods will be suitable for
> > non-programmers, but the use of Jython scripts follows the OFBiz
> > pattern of providing simple methods for simple tasks, but making it
> > easy to drop down to the level needed to solve the problem. At some
> > point (right away?), it will be necessary to allow complex joing of
> > tests with AND, OR and NOT operators.
> > Add Test Overlay Config File One of the modifications to the OOTB test
> > script generated by TCPProxy will be that the script will look for the
> > existence a file path as a parameters and use it to overwrite the
> > default test script values (ie. the values typed in by the user when
> > the script was being generated). The script could have multiple levels
> > of complex scripts, but eventually, they must call one of the
> > "request" tests. Keep in mind that the base test script could have a
> > large number of "request" scripts, but they would not all need to be
> > used by the user-defined config file; it could just use a subset. So
> > the base test script may not, in fact, be totally generated by one
> > session of user interaction with the system under test; it could be
> > built up over time as new functionality is added without having to run
> > thru the test. The OOTB behavior is to sequentially number the tests,
> > but in this case, we may wish to name them with the request name that
> > they interact with.
> >
> > These high level test scripts would need "include" functionality so
> > that they could used standard building blocks.
> > Source Code Control to Handle Iterations
> > Appendices
> > Standard Generated Grinder Test Script This is the actual output from
> > a TCPProxy session:
> >
> > # The Grinder 3.0-beta30
> > # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM
> >
> > from net.grinder.script import Test
> > from net.grinder.script.Grinder import grinder
> > from net.grinder.plugin.http
> > import HTTPPluginControl, HTTPRequest
> > from HTTPClient import NVPair
> > connectionDefaults = HTTPPluginControl.getConnectionDefaults()
> > httpUtilities = HTTPPluginControl.getHTTPUtilities()
> >
> > # To use a proxy server, uncomment the next line and set the host
> > and port.
> >
> > # connectionDefaults.setProxyServer("localhost", 8001)
> >
> > # These definitions at the top level of the file are evaluated once,
> > # when the worker process is started.
> >
> > connectionDefaults.defaultHeaders
> > = \
> > ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1;
> > en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'),
> > NVPair('Accept-Encoding', 'gzip,deflate'),
> > NVPair('Accept-Language', 'en-us,en;q=
> > 0.5'),
> > NVPair('Accept-Charset', 'UTF-8,*'),
> > NVPair('Accept',
> > 'text/xml,application/xml,application/xhtml+xml,text/html;q=
> > 0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), )
> >
> > headers0= \
> > ( )
> >
> > headers1= \
> >
> > ( NVPair('Referer', 'https://localhost:8443/webtools/control/
> > main' ), )
> >
> > headers2= \
> > ( NVPair('Referer', '
> > https://localhost:8443/webtools/control/checkLogin/main'), )
> >
> > url0 = ' https://localhost:8443'
> >
> > # Create an HTTPRequest for each request, then replace the
> > # reference to the HTTPRequest with an instrumented version.
> >
> > # You can access the unadorned instance using request101.__target__.
> > request101 = HTTPRequest(url=url0, headers=headers0)
> > request101 = Test(101, 'GET /').wrap(request101)
> >
> > request102 = HTTPRequest(url=url0, headers=headers0)
> >
> > request102 = Test(102, 'GET main').wrap(request102)
> >
> > request201 = HTTPRequest(url=url0, headers=headers1)
> > request201 = Test(201, 'GET main').wrap(request201)
> >
> > request301 = HTTPRequest(url=url0, headers=headers2)
> >
> > request301 = Test(301, 'POST login').wrap(request301)
> >
> >
> > class TestRunner:
> > """A TestRunner instance is created for each worker thread."""
> >
> > # A method for each recorded page.
> >
> > def page1(self):
> > """GET / (requests 101-102)."""
> >
> > # Expecting 302 'Moved Temporarily'
> > result = request101.GET('/webtools/')
> >
> > grinder.sleep(16)
> >
> > request102.GET('/webtools/control/main')
> >
> > return result
> >
> > def page2(self):
> > """GET main (request 201)."""
> > result = request201.GET('/webtools/control/checkLogin/main')
> >
> >
> > return result
> >
> > def page3(self):
> > """POST login (request 301)."""
> > result = request301.POST('/webtools/control/login',
> > ( NVPair('USERNAME', 'admin'),
> >
> > NVPair('PASSWORD', 'ofbiz'), ),
> > ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), ))
> >
> > return result
> >
> > def __call__(self):
> > """This method is called for every run performed by the worker
> > thread."""
> >
> > self.page1() # GET / (requests 101-102)
> >
> > grinder.sleep(4328)
> > self.page2() # GET main (request 201)
> >
> > grinder.sleep(2109)
> > self.page3() # POST login (request 301)
> >
> >
> >
> > def instrumentMethod(test, method_name, c=TestRunner):
> > """Instrument a method with the given Test."""
> > unadorned = getattr(c, method_name)
> > import new
> > method = new.instancemethod
> > (test.wrap(unadorned), None, c)
> > setattr(c, method_name, method)
> >
> > # Replace each method with an instrumented version.
> > # You can call the unadorned method using self.page1.__target__().
> > instrumentMethod(Test(100, 'Page 1'), 'page1')
> >
> > instrumentMethod(Test(200, 'Page 2'), 'page2')
> > instrumentMethod(Test(300, 'Page 3'), 'page3')
> >
> >
> >
> > Possible Custom Generated Grinder Test Script # The Grinder 3.0-beta30
> > # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM
> >
> > from net.grinder.script import Test
> > from net.grinder.script.Grinder import grinder
> > from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest
> >
> > from HTTPClient import NVPair
> > import AGrinderTest
> >
> > connectionDefaults = HTTPPluginControl.getConnectionDefaults()
> > httpUtilities = HTTPPluginControl.getHTTPUtilities()
> >
> > # To use a proxy server, uncomment the next line and set the host
> > and port.
> >
> > # connectionDefaults.setProxyServer("localhost", 8001)
> >
> > # These definitions at the top level of the file are evaluated once,
> > # when the worker process is started.
> >
> > connectionDefaults.defaultHeaders
> > = \
> > ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1;
> > en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'),
> > NVPair('Accept-Encoding', 'gzip,deflate'),
> > NVPair('Accept-Language', 'en-us,en;q=
> > 0.5'),
> > NVPair('Accept-Charset', 'UTF-8,*'),
> > NVPair('Accept',
> > 'text/xml,application/xml,application/xhtml+xml,text/
> > html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'),
> > )
> >
> > headers0= \
> > ( )
> >
> > headers1= \
> >
> > ( NVPair('Referer', 'https://localhost:8443/webtools/control/
> > main'), )
> >
> > headers2= \
> > ( NVPair('Referer', '
> > https://localhost:8443/webtools/control/checkLogin/main'), )
> >
> > url0 = 'https://localhost:8443'
> >
> > # Create an HTTPRequest for each request, then replace the
> > # reference to the HTTPRequest with an instrumented version.
> >
> > # You can access the unadorned instance using request101.__target__.
> > request101 = HTTPRequest(url=url0, headers=headers0)
> > request101 = Test(101, 'GET /').wrap(request101)
> >
> > request102 = HTTPRequest(url=url0, headers=headers0)
> >
> > request102 = Test(102, 'GET main').wrap(request102)
> >
> > request201 = HTTPRequest(url=url0, headers=headers1)
> > request201 = Test(201, 'GET main').wrap(request201)
> >
> > request301 = HTTPRequest(url=url0, headers=headers2)
> >
> > request301 = Test(301, 'POST login').wrap(request301)
> >
> >
> > class TestRunner:
> > """A TestRunner instance is created for each worker thread."""
> >
> > # A method for each recorded page.
> >
> > def page1(self):
> > """GET / (requests 101-102)."""
> >
> > # Expecting 302 'Moved Temporarily'
> > result = request101.GET('/webtools/')
> >
> > grinder.sleep(16)
> >
> > request102.GET('/webtools/control/main')
> >
> > return result
> >
> > def page2(self):
> > """GET main (request 201)."""
> > result = request201.GET('/webtools/control/checkLogin/main')
> >
> >
> > return result
> >
> > def page3(self):
> > """POST login (request 301)."""
> > result = request301.POST('/webtools/control/login',
> > ( NVPair('USERNAME', 'admin'),
> >
> > NVPair('PASSWORD', 'ofbiz'), ),
> > ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), ))
> >
> > return result
> >
> >
> > """This is not working code. Just an idea of how the generated
> > code would look"""
> >
> > def webtoolsWrap(self, overlayMap):
> > inMap = {}
> > outMap = {}
> > self.agrinder.runWrappedTest(page1, inMap, outMap, overlayMap)
> >
> > def mainWrap(self, overlayMap):
> > inMap = {}
> > outMap = {}
> >
> > self.agrinder.runWrappedTest(page2, inMap, outMap, overlayMap)
> >
> > def loginWrap(self, overlayMap):
> > inMap = {'USERNAME':'admin', 'PASSWORD':'ofbiz'}
> > outMap = {}
> > self.agrinder.runWrappedTest
> > (page2, inMap, outMap, overlayMap)
> >
> > def __call__(self):
> > """This method is called for every run performed by the worker
> > thread."""
> >
> > """I am going to do some handwaving here because I want to get
> > this doc out today,
> >
> > but there would be code here to read in an argument from the
> > command line and
> > use it as a file path to read in the "test overlay" XML doc"""
> > self.agrinder = AGrinderTest.AGrinder
> > ()
> > testNode = self.agrinder.getOverlayTest("webtools", sys.argv)
> > self.webtoolsWrap(testNode) # GET / (requests 101-102)
> >
> > grinder.sleep(4328)
> > self.mainWrap(testNode) # GET main (request 201)
> >
> >
> > grinder.sleep(2109)
> > self.loginWrap(testNode) # POST login (request 301)
> >
> >
> > def instrumentMethod(test, method_name, c=TestRunner):
> > """Instrument a method with the given Test."""
> >
> > unadorned = getattr(c, method_name)
> > import new
> > method = new.instancemethod(test.wrap(unadorned), None, c)
> > setattr(c, method_name, method)
> >
> > # Replace each method with an instrumented version.
> >
> > # You can call the unadorned method using self.page1.__target__().
> > instrumentMethod(Test(100, 'Page 1'), 'page1')
> > instrumentMethod(Test(200, 'Page 2'), 'page2')
> > instrumentMethod(Test(300, 'Page 3'), 'page3')
> >
> >
> > Possible Overlay XML File This example is probably not even close to
> > what the final test overlay XML script will look like, but I think it
> > is better to have something to go from. <tests>
> > <test name="webtools">
> > <subtest name="login">
> > <input>
> > <field name="USERNAME">jdoe</field>
> > <field name="PASSWORD">id9Ed3jk</field>
> >
> > </input>
> > <match>
> > <field type="regex" op="not">not found</field>
> > <field type="script" op="true">isSuccess()</field>
> >
> > </match>
> > </subtest>
> > </test>
> > </tests>
>
> Best Regards,
>
> Si
> [EMAIL PROTECTED]
>
>
>
>
>
