Yes I agree with Al, having both is surely better (of course more work also...)
Jacques
From: "Al Byers" <[EMAIL PROTECTED]>
> Si,
>
> Thanks for the feedback. I debated over the xml language. I think it
> is important to involve a different level of user. But the safe thing
> to do would be to create the api that you mention and then wrap it in
> an xml layer later on if there is demand.
>
> -Al
>
> On 11/2/06, Si Chen <[EMAIL PROTECTED]> wrote:
> > Al,
> >
> > This is great that you're doing this. I think a testing framework is
> > really what we need right now.
> >
> > My suggestion is instead of creating an xml language like you show at
> > the bottom to make some API methods that could be called in Jython
> > scripts. I think that might be easier for people to work with and
> > would require less work over time in maintaining and extending the
> > xml language.
> >
> > On Nov 1, 2006, at 10:49 AM, Al Byers wrote:
> >
> > > I am having trouble getting my messages posted to the list. This one
> > > is taking a long time and I wonder if it is because it has HTML in the
> > > content and an attachment. I am pasting it in plain test.
> > > ...
> > > One of the presentations at the Users Conference will be me talking on
> > > my experiences with Grinder. With David there, we hope to go far past
> > > that to discuss strategies for adding testing functionality to OFBiz.
> > > I have no experience that qualifies me to be an expert in this area -
> > > just have a need. Rather than just wait for everyone to show up at the
> > > conference, I felt it would be helpful to try to treat some of the
> > > subjects ahead of time. Of course, not everything will be decided at
> > > the conference, so as much input as possible from everyone will be
> > > needed. Also, if there is anyone with strong Python and/or Grinder
> > > skills, their help would also be appreciated.
> > >
> > >
> > > OFBiz Testing Initiative
> > > GOAL
> > > BACKGROUND
> > > Problem with Design Documents
> > > Applicable Features of OFBiz
> > > Applicable Features of Grinder
> > > Applicable Features of Extreme Programming
> > > REQUIREMENTS
> > > Easy Generation
> > > Handle Complex Tests
> > > Adaptable per Application
> > > Rugged
> > > Handle Iterations
> > > Usable at Multiple Proficiency Levels
> > > Simplify Output Testing
> > > STRAWMAN PLAN
> > > Modify Test Script Generation
> > > Add Test Overlay Config File
> > > Source Code Control to Handle Iterations
> > > Appendices
> > > Standard Generated Grinder Test Script
> > > Possible Custom Generated Grinder Test Script
> > > Possible Overlay XML File
> > > GOAL The goal of the OFBiz Testing Initiative (OFBTI) is to add
> > > functionality to OFBiz that will streamline the functional testing
> > > process to the point that it will be cost-effective and beneficial to
> > > write comprehensive tests. Ideally, the tools could be used by
> > > non-programmers.
> > >
> > > A secondary goal would be to add tools to make design specs easier to
> > > write and more useful.
> > > BACKGROUND
> > > Problem with Design Documents Design documents, though necessary, have
> > > serious drawbacks.
> > > Expensive to create - should probably take as much time as the
> > > implementation.
> > >
> > > Never complete enough - always need to go back to client, anyway.
> > >
> > > No automation help in implementation and validation - as text
> > > documents, they must be manually translated into architecture
> > > documents.
> > >
> > > Easy to get out-of-date - while many times they are supposed to be
> > > "living" documents, in reality, they almost never are.
> > >
> > >
> > > Applicable Features of Extreme Programming Extreme programming
> > > addresses many of the shortcomings of the more conventional design
> > > document approach. Instead of a complete design doc or even use case
> > > up front, all that is required is a "story". The story needs only be
> > > complete enough to generate the first test and the first test is
> > > usually just a placeholder. Then in the course of many iterations, the
> > > functional test is enhanced, not the design document. The tests are
> > > living and serve the useful purpose of verifying that things have not
> > > broken as code gets changed.
> > >
> > > One of the very useful things about this approach is that when bugs
> > > are discovered, a test or subtest is added to guard against it ever
> > > creeping back into the code.
> > > Applicable Features of OFBiz OFBiz is different from other development
> > > environments, and it would be good to identify the features that could
> > > be used advantageously or which must be dealt with.
> > >
> > > OFBiz is highly configurable via XML formatted files and it would be
> > > good to continue that pattern. XML config files, because they have
> > > associated schema, make it feasible for non-programmers and those not
> > > intimately familiar with OFBiz to make changes and perform certain
> > > programming tasks.
> > >
> > > OFBiz has a general pattern of offering easily configurable options
> > > via data parameters, but always offering easy access to the lowest
> > > level of programming for those needed cases without requiring a huge
> > > environment adaptation. For instance, the screen widget config files
> > > allow screens to be created with miminal data and allow the user to be
> > > prompted via the schema in an XML editor, but if there is a construct
> > > that the widgets do not handle, the user can just throw in a call to
> > > an HTML component or a FreeMarker template - they do not need to
> > > abandon the screen widget framework.
> > >
> > > OFBiz is a service oriented framework and there are many tools that
> > > can be used aid in that process. If a request is processed by a
> > > service (in lieu of an event) then the input parameters can be
> > > automatically taken from the HTTP stream and automatically converted
> > > to the right type. Also, the form widget can build forms from the
> > > service definition.
> > > Applicable Features of Grinder There are multiple options for testing
> > > frameworks, but Grinder offers the following unique advantages.
> > > A robust test script generator - TCPProxy attempts to assign return
> > > parameters to variables and reuse them, rather than passing
> > > literals around. This means that it does not instantly break when keys
> > > are autogenerated.
> > > Grinder has convenient customization point - script generation is done
> > > via an XSLT stylesheet that makes use of Java extensions. This would
> > > be a natural place to make modifications to suit OFBTI. There are also
> > > filters for the requests and responses that can be swapped out via
> > > command line parameters.
> > > Jython - allows seamless switching between a highly productive
> > > scripting environment and regular Java code. Because of this feature,
> > > Jython has a lot of possibilities within OFBiz.
> > > Integrated functional and jUnit testing - the use of Jython allows for
> > > the same environment to be used to run HTTP client system tests and
> > > jUnit functional tests.
> > > REQUIREMENTS
> > > Easy Generation The use of a test script generator, such as Grinder's
> > > TCPProxy, allows test scripts to be created by capturing the input of
> > > a user at a browser.
> > > Handle Complex Tests A further enhancement would be to allow the
> > > creation of scripts by supplying a few data parameters or by easily
> > > modifying automatically generated scripts. One of the biggest
> > > drawbacks of end user testing environments is that when something
> > > changes, the associated test script is usually unusable. A big
> > > improvement would be to allow tests to be created by chaining together
> > > subtests.
> > > Adaptable per Application Many applications will have special
> > > characteristics that need to be handled specially. In one instance, I
> > > found that Grinder was using the '$' character to form Python variable
> > > names (which doesn't work) because that is how the HTTP parameters
> > > were named.
> > > Rugged
> > > Handle Iterations The general idea of XP is that the tests become more
> > > complex and comprehensive as the code becomes more complete. It would
> > > probably be a good idea to retain visibility to past test setups -
> > > though, at this time, I am not sure why.
> > > Usable at Multiple Proficiency Levels It would be good to have the
> > > testing environment useable, or at least understandable, by
> > > non-programmers. JUnit tests would not meet this requirement. Neither
> > > would Python scripts (though it could come close). For project
> > > managers, an XML-based configuration environment would be needed.
> > > Simplify Output Testing The analysis of the HTML pages that are
> > > returned by Grinder tests is one of the more problematic areas of
> > > end-user testing. The more common scenarios would need to be handled
> > > by the XML-based configuration environment mentioned above. The
> > > foreseeable cases would be matching of form values, existence or
> > > non-existence anywhere of a phrase or something that can be tested by
> > > an XPath expression.
> > > STRAWMAN PLAN What is discussed below is how I see tackling the
> > > testing problem. I am only interested in what will be supported by the
> > > community, so it can be changed.
> > >
> > > The overall approach is to use as much of Grinder as possible, and
> > > allow for the generated Grinder test scripts to be overwritten by a
> > > user generated XML test configuration file.
> > > Modify Test Script Generation The Grinder TCPProxy program will be
> > > used, but the XSLT stylesheet and the associated Java extension
> > > classes will be enhanced. The main change will be that the individual
> > > request tests that Grinder generates for each round trip to the server
> > > will be wrapped by code that will check to see if there are data
> > > overrides to be made coming from the user generated high level test
> > > config file. Each wrapper will also have a dictionary (ie. map) that
> > > defines tests to be made on each low level result. These also will be
> > > modified by the higher level config file. If there is not higher level
> > > config file, the script as generated by TCPProxy will run with no
> > > modifications.
> > > Input Dictionary The input directory will do more than just allow the
> > > user to supply literal values as test input. There will be helper
> > > functions for allowing the user to randomly pick values from a list.
> > > Also there will be helper functions for generating reasonably looking
> > > addresses. The user will be able to create Python scripts to generated
> > > special input. See the appendix for samples of what this would look
> > > like.
> > > Output Dictionary The output dictionary, which contains the test
> > > criteria, would have test helper functions of the following sort:
> > > Literal value
> > > Regular expression
> > > Form value
> > > XPath
> > > Custom Jython scripts Many of the test methods will be suitable for
> > > non-programmers, but the use of Jython scripts follows the OFBiz
> > > pattern of providing simple methods for simple tasks, but making it
> > > easy to drop down to the level needed to solve the problem. At some
> > > point (right away?), it will be necessary to allow complex joing of
> > > tests with AND, OR and NOT operators.
> > > Add Test Overlay Config File One of the modifications to the OOTB test
> > > script generated by TCPProxy will be that the script will look for the
> > > existence a file path as a parameters and use it to overwrite the
> > > default test script values (ie. the values typed in by the user when
> > > the script was being generated). The script could have multiple levels
> > > of complex scripts, but eventually, they must call one of the
> > > "request" tests. Keep in mind that the base test script could have a
> > > large number of "request" scripts, but they would not all need to be
> > > used by the user-defined config file; it could just use a subset. So
> > > the base test script may not, in fact, be totally generated by one
> > > session of user interaction with the system under test; it could be
> > > built up over time as new functionality is added without having to run
> > > thru the test. The OOTB behavior is to sequentially number the tests,
> > > but in this case, we may wish to name them with the request name that
> > > they interact with.
> > >
> > > These high level test scripts would need "include" functionality so
> > > that they could used standard building blocks.
> > > Source Code Control to Handle Iterations
> > > Appendices
> > > Standard Generated Grinder Test Script This is the actual output from
> > > a TCPProxy session:
> > >
> > > # The Grinder 3.0-beta30
> > > # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM
> > >
> > > from net.grinder.script import Test
> > > from net.grinder.script.Grinder import grinder
> > > from net.grinder.plugin.http
> > > import HTTPPluginControl, HTTPRequest
> > > from HTTPClient import NVPair
> > > connectionDefaults = HTTPPluginControl.getConnectionDefaults()
> > > httpUtilities = HTTPPluginControl.getHTTPUtilities()
> > >
> > > # To use a proxy server, uncomment the next line and set the host
> > > and port.
> > >
> > > # connectionDefaults.setProxyServer("localhost", 8001)
> > >
> > > # These definitions at the top level of the file are evaluated once,
> > > # when the worker process is started.
> > >
> > > connectionDefaults.defaultHeaders
> > > = \
> > > ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1;
> > > en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'),
> > > NVPair('Accept-Encoding', 'gzip,deflate'),
> > > NVPair('Accept-Language', 'en-us,en;q=
> > > 0.5'),
> > > NVPair('Accept-Charset', 'UTF-8,*'),
> > > NVPair('Accept',
> > > 'text/xml,application/xml,application/xhtml+xml,text/html;q=
> > > 0.9,text/plain;q=0.8,image/png,*/*;q=0.5'), )
> > >
> > > headers0= \
> > > ( )
> > >
> > > headers1= \
> > >
> > > ( NVPair('Referer', 'https://localhost:8443/webtools/control/
> > > main' ), )
> > >
> > > headers2= \
> > > ( NVPair('Referer', '
> > > https://localhost:8443/webtools/control/checkLogin/main'), )
> > >
> > > url0 = ' https://localhost:8443'
> > >
> > > # Create an HTTPRequest for each request, then replace the
> > > # reference to the HTTPRequest with an instrumented version.
> > >
> > > # You can access the unadorned instance using request101.__target__.
> > > request101 = HTTPRequest(url=url0, headers=headers0)
> > > request101 = Test(101, 'GET /').wrap(request101)
> > >
> > > request102 = HTTPRequest(url=url0, headers=headers0)
> > >
> > > request102 = Test(102, 'GET main').wrap(request102)
> > >
> > > request201 = HTTPRequest(url=url0, headers=headers1)
> > > request201 = Test(201, 'GET main').wrap(request201)
> > >
> > > request301 = HTTPRequest(url=url0, headers=headers2)
> > >
> > > request301 = Test(301, 'POST login').wrap(request301)
> > >
> > >
> > > class TestRunner:
> > > """A TestRunner instance is created for each worker thread."""
> > >
> > > # A method for each recorded page.
> > >
> > > def page1(self):
> > > """GET / (requests 101-102)."""
> > >
> > > # Expecting 302 'Moved Temporarily'
> > > result = request101.GET('/webtools/')
> > >
> > > grinder.sleep(16)
> > >
> > > request102.GET('/webtools/control/main')
> > >
> > > return result
> > >
> > > def page2(self):
> > > """GET main (request 201)."""
> > > result = request201.GET('/webtools/control/checkLogin/main')
> > >
> > >
> > > return result
> > >
> > > def page3(self):
> > > """POST login (request 301)."""
> > > result = request301.POST('/webtools/control/login',
> > > ( NVPair('USERNAME', 'admin'),
> > >
> > > NVPair('PASSWORD', 'ofbiz'), ),
> > > ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), ))
> > >
> > > return result
> > >
> > > def __call__(self):
> > > """This method is called for every run performed by the worker
> > > thread."""
> > >
> > > self.page1() # GET / (requests 101-102)
> > >
> > > grinder.sleep(4328)
> > > self.page2() # GET main (request 201)
> > >
> > > grinder.sleep(2109)
> > > self.page3() # POST login (request 301)
> > >
> > >
> > >
> > > def instrumentMethod(test, method_name, c=TestRunner):
> > > """Instrument a method with the given Test."""
> > > unadorned = getattr(c, method_name)
> > > import new
> > > method = new.instancemethod
> > > (test.wrap(unadorned), None, c)
> > > setattr(c, method_name, method)
> > >
> > > # Replace each method with an instrumented version.
> > > # You can call the unadorned method using self.page1.__target__().
> > > instrumentMethod(Test(100, 'Page 1'), 'page1')
> > >
> > > instrumentMethod(Test(200, 'Page 2'), 'page2')
> > > instrumentMethod(Test(300, 'Page 3'), 'page3')
> > >
> > >
> > >
> > > Possible Custom Generated Grinder Test Script # The Grinder 3.0-beta30
> > > # HTTP script recorded by TCPProxy at Oct 31, 2006 5:44:23 AM
> > >
> > > from net.grinder.script import Test
> > > from net.grinder.script.Grinder import grinder
> > > from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest
> > >
> > > from HTTPClient import NVPair
> > > import AGrinderTest
> > >
> > > connectionDefaults = HTTPPluginControl.getConnectionDefaults()
> > > httpUtilities = HTTPPluginControl.getHTTPUtilities()
> > >
> > > # To use a proxy server, uncomment the next line and set the host
> > > and port.
> > >
> > > # connectionDefaults.setProxyServer("localhost", 8001)
> > >
> > > # These definitions at the top level of the file are evaluated once,
> > > # when the worker process is started.
> > >
> > > connectionDefaults.defaultHeaders
> > > = \
> > > ( NVPair('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1;
> > > en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7'),
> > > NVPair('Accept-Encoding', 'gzip,deflate'),
> > > NVPair('Accept-Language', 'en-us,en;q=
> > > 0.5'),
> > > NVPair('Accept-Charset', 'UTF-8,*'),
> > > NVPair('Accept',
> > > 'text/xml,application/xml,application/xhtml+xml,text/
> > > html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'),
> > > )
> > >
> > > headers0= \
> > > ( )
> > >
> > > headers1= \
> > >
> > > ( NVPair('Referer', 'https://localhost:8443/webtools/control/
> > > main'), )
> > >
> > > headers2= \
> > > ( NVPair('Referer', '
> > > https://localhost:8443/webtools/control/checkLogin/main'), )
> > >
> > > url0 = 'https://localhost:8443'
> > >
> > > # Create an HTTPRequest for each request, then replace the
> > > # reference to the HTTPRequest with an instrumented version.
> > >
> > > # You can access the unadorned instance using request101.__target__.
> > > request101 = HTTPRequest(url=url0, headers=headers0)
> > > request101 = Test(101, 'GET /').wrap(request101)
> > >
> > > request102 = HTTPRequest(url=url0, headers=headers0)
> > >
> > > request102 = Test(102, 'GET main').wrap(request102)
> > >
> > > request201 = HTTPRequest(url=url0, headers=headers1)
> > > request201 = Test(201, 'GET main').wrap(request201)
> > >
> > > request301 = HTTPRequest(url=url0, headers=headers2)
> > >
> > > request301 = Test(301, 'POST login').wrap(request301)
> > >
> > >
> > > class TestRunner:
> > > """A TestRunner instance is created for each worker thread."""
> > >
> > > # A method for each recorded page.
> > >
> > > def page1(self):
> > > """GET / (requests 101-102)."""
> > >
> > > # Expecting 302 'Moved Temporarily'
> > > result = request101.GET('/webtools/')
> > >
> > > grinder.sleep(16)
> > >
> > > request102.GET('/webtools/control/main')
> > >
> > > return result
> > >
> > > def page2(self):
> > > """GET main (request 201)."""
> > > result = request201.GET('/webtools/control/checkLogin/main')
> > >
> > >
> > > return result
> > >
> > > def page3(self):
> > > """POST login (request 301)."""
> > > result = request301.POST('/webtools/control/login',
> > > ( NVPair('USERNAME', 'admin'),
> > >
> > > NVPair('PASSWORD', 'ofbiz'), ),
> > > ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), ))
> > >
> > > return result
> > >
> > >
> > > """This is not working code. Just an idea of how the generated
> > > code would look"""
> > >
> > > def webtoolsWrap(self, overlayMap):
> > > inMap = {}
> > > outMap = {}
> > > self.agrinder.runWrappedTest(page1, inMap, outMap, overlayMap)
> > >
> > > def mainWrap(self, overlayMap):
> > > inMap = {}
> > > outMap = {}
> > >
> > > self.agrinder.runWrappedTest(page2, inMap, outMap, overlayMap)
> > >
> > > def loginWrap(self, overlayMap):
> > > inMap = {'USERNAME':'admin', 'PASSWORD':'ofbiz'}
> > > outMap = {}
> > > self.agrinder.runWrappedTest
> > > (page2, inMap, outMap, overlayMap)
> > >
> > > def __call__(self):
> > > """This method is called for every run performed by the worker
> > > thread."""
> > >
> > > """I am going to do some handwaving here because I want to get
> > > this doc out today,
> > >
> > > but there would be code here to read in an argument from the
> > > command line and
> > > use it as a file path to read in the "test overlay" XML doc"""
> > > self.agrinder = AGrinderTest.AGrinder
> > > ()
> > > testNode = self.agrinder.getOverlayTest("webtools", sys.argv)
> > > self.webtoolsWrap(testNode) # GET / (requests 101-102)
> > >
> > > grinder.sleep(4328)
> > > self.mainWrap(testNode) # GET main (request 201)
> > >
> > >
> > > grinder.sleep(2109)
> > > self.loginWrap(testNode) # POST login (request 301)
> > >
> > >
> > > def instrumentMethod(test, method_name, c=TestRunner):
> > > """Instrument a method with the given Test."""
> > >
> > > unadorned = getattr(c, method_name)
> > > import new
> > > method = new.instancemethod(test.wrap(unadorned), None, c)
> > > setattr(c, method_name, method)
> > >
> > > # Replace each method with an instrumented version.
> > >
> > > # You can call the unadorned method using self.page1.__target__().
> > > instrumentMethod(Test(100, 'Page 1'), 'page1')
> > > instrumentMethod(Test(200, 'Page 2'), 'page2')
> > > instrumentMethod(Test(300, 'Page 3'), 'page3')
> > >
> > >
> > > Possible Overlay XML File This example is probably not even close to
> > > what the final test overlay XML script will look like, but I think it
> > > is better to have something to go from. <tests>
> > > <test name="webtools">
> > > <subtest name="login">
> > > <input>
> > > <field name="USERNAME">jdoe</field>
> > > <field name="PASSWORD">id9Ed3jk</field>
> > >
> > > </input>
> > > <match>
> > > <field type="regex" op="not">not found</field>
> > > <field type="script" op="true">isSuccess()</field>
> > >
> > > </match>
> > > </subtest>
> > > </test>
> > > </tests>
> >
> > Best Regards,
> >
> > Si
> > [EMAIL PROTECTED]
