Jeremiah, Thanks for so detailed explanation.
As a side note, did you see Taurus project (http://gettaurus.org/), which tries to bring some YAML-based way to express JMeter test plans? Andrey Pokhilko On 08/10/2016 08:49 PM, Epp, Jeremiah W (Contractor) wrote: >> -----Original Message----- >> From: Philippe Mouawad [mailto:[email protected]] >> Sent: Thursday, July 28, 2016 4:37 PM >> To: [email protected] >> Subject: Re: JMeter DSL discussion >> >> I think DSL has 2 important advantages: >> - It is more readable in source repositories. >> - it is better for developers who load test their application in CI/CD. >> This is clearly an important move JMeter must follow. > If I might chime in on this conversation? I'd like to offer the > perspective of an organisation actually using JMeter with CI right now. > > To preface, we're doing something rather unusual in that we're actually > generating test plans in CI from our functional tests using a command-line > recorder daemon I wrote (based on JMeter; I'm still trying to get permission > to release the source) and then correlating them with post-processing > scripts. > > In this capacity, we would be VERY interested in the ability to record to > and operate on a format that is not JMX. From our perspective, a robust and > well-considered test plan DSL brings two major advantages to the table: > determinism and structural soundness. > > By determinism, I mean that every node of the test plan tree is consistently > represented. By structural soundness, I mean that every relationship > between nodes arises as a result of the structure expressed in the test plan > file itself (i.e. implementation details don't create implicit structure). > > To elaborate further, currently, test plans are written to JMX files. JMX > is an XML-based format which, if you look at its actual structure, is > moderately insane, horrendously verbose, and basically undocumented. After > several months of working with it, I believe it's better to consider it a > serialisation of a Java data structure that happens to be in text form more > than an actual file format. I would personally love nothing more than to > see it deprecated. > > If you haven't looked recently, I'll recap: most testElements use stringProp > and friends with a name property to configure their nodes, and the > parent/child relationship between most elements is implicit, relying on the > hashTree immediately following it in the file. The actual tree-like nature > of an XML document is largely ignored in many cases within the format. > (Also, again, it's XML, so there are invalid characters, which is bad.) > > For example, this is the structure of a single HTTP request with one > Argument (and some of my own comments): > > <HTTPSamplerProxy guiclass="HttpTestSampleGui" > testclass="HTTPSamplerProxy" testname="9952 /as/authorization.oauth2" > enabled="true"> > <elementProp name="HTTPsampler.Arguments" elementType="Arguments" > guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true"> > <collectionProp name="Arguments.arguments"> > <elementProp name="state" elementType="HTTPArgument"> <!-- Note > the redundancies, particularly the names --> > <boolProp name="HTTPArgument.always_encode">false</boolProp> > <stringProp name="Argument.name">state</stringProp> > <stringProp name="Argument.value">${state009951}</stringProp> > <stringProp name="Argument.metadata">=</stringProp> > <boolProp name="HTTPArgument.use_equals">true</boolProp> > </elementProp> > </collectionProp> > </elementProp> > <stringProp /> <!-- I'll omit all the attributes and values from > here onward; I think you get the idea --> > <!-- Nine more stringProp here --> > <boolProp /> > <!-- Four more boolProp here --> > </HTTPSamplerProxy> > <hashTree> > <HeaderManager> > <collectionProp> > <elementProp> > <stringProp /> > <stringProp /> > </elementProp> > <!-- Four more elementProp with two stringProp each here --> > </collectionProp> > </HeaderManager> > <hashTree/> <!-- This empty hashTree is NOT optional --> > <RegexExtractor> > <stringProp /> > <!-- Six more stringProp here --> > </RegexExtractor> > <hashTree/> <!-- This empty hashTree at the end IS optional --> > </hashTree> > > Unaltered, this single request description is 107 lines and 7208 bytes. > > Note the <hashTree> after the </HTTPSamplerProxy> (and many -- but not all! > -- other elements) is understood by JMeter as containing the _children_ of > the <HTTPSamplerProxy>. For elements with no children, it's still necessary > that they be followed by an empty <hashTree/> if there are more elements at > the same level of the tree. (This is, I believe, a consequence of a > TestPlan's internal representation as a ListedHashTree of ListedHashTrees.) > As far as we've been able to determine, this format is entirely > undocumented. > > The consequence of all this is it is EXTREMELY difficult to reliably operate > on a JMX file with external tools, and much of it has been trial-and-error. > Really, it's been a nightmare. A DSL should have clean and clear semantics > that prevent parsing ambiguity and speed development time. Anything that > does not is a wasted effort. > > So bearing all that in mind, I would strongly recommend this additional > requirement: a new DSL MUST decouple the test plan format from > implementation details. A good example of how this is bad in JMX is all the > attribute noise on any sampler-- the test plan shouldn't have to know about > the guiclass, testclass, elementType, etc. It's all obvious from the > structure and the type of the node. > >> - the language it should be developed in. My personal preference would go >> for Groovy but I am not an expert in DSLs. > Hang on, why would you develop your DSL in a different language from what > JMeter itself is actually written in? This wold seem to add an entire new > toolchain to the build process. > >> - how to express what today we can do through JSR223 languages which might >> be the most complex think to translate > Is this really an issue? This seems orthogonal to the goal of creating a > language to describe _test plans_ themselves. As long as this DSL has a way > to express the JSR223 sampler, saving the actual code that goes in it should > go without saying. > > Regards, > Wyatt > > Confidentiality Notice: This electronic message transmission, including any > attachment(s), may contain confidential, proprietary, or privileged > information from Chemical Abstracts Service ("CAS"), a division of the > American Chemical Society ("ACS"). If you have received this transmission in > error, be advised that any disclosure, copying, distribution, or use of the > contents of this information is strictly prohibited. Please destroy all > copies of the message and contact the sender immediately by either replying > to this message or calling 614-447-3600. >
