Good work Branden, You gathered the information and came to a conclusion: Developers are going to maintain the performance tests. In general, developers prefer to maintain API's rather than use GUI tools and Jmeter fragments.
To avoid duplication of effort, I will build up my Tsung skills and later see where I can help. Alan Alan Berg Group Education and Research Services Central Computer Services University of Amsterdam ________________________________________ From: [email protected] [[email protected]] on behalf of Branden Visser [[email protected]] Sent: 30 July 2012 13:34 To: [email protected] list Subject: Re: [oae-dev] Load testing tool Hi everyone, Just providing an update on this. Alan was able to pull together the unit tests outlined previously [1], and I was able to do so using the Tsung-based Open-Performance-Automation-Framework (OPAF) [2]. I'll speak to the following points with my experience with OPAF: >> 1. How easy is it to create a re-usable test case. Is it feasible to >> think that an institution will pull the components together needed to >> contribute back to the effort? To create re-usable components in the OPAF, I had to add new elements to the testing API using others as examples. With my limited Ruby experience, aside from a few bumps associated to my learning curve, this was relatively painless. The good thing was that once I added an element to the API, I could use it in many different test cases without having to know anything about the requests being fired. I think that it would be rather easy for a developer to create re-usable test-cases and API components that are of value to the community. >> 2. Which is more feasible for the *test maintainers* to keep updated? >> I highlight test maintainers, because at the end of the day, if those >> who will ultimately be responsible for maintaining the tests are not >> comfortable with the medium, I think the effort will fail First, I believe the responsibility to maintain the performance tests will ultimately fall on to the developers, or those with development background. That said I think it is most feasible for me as a developer to maintain the OPAF APIs instead of the JMeter scripts. While creating the OPAF test cases, I managed to update out-dated test cases (pre-1.3.0) quite easily, and those updates propagate to other test suites automatically. In general, I found that working with the tests through an API made things more explicit, which allowed me to make changes quite quickly. While I think a similar system could be put in place with JMeter using fragments, it does require special knowledge of JMeter, and I've never seen it carried out successfully within the community. >> 3. What effort is required to perform the tests and get the data into >> a valuable format -- is it feasible to do weekly? nightly? I am still in the process of getting a proof-of-concept for Tsung, but I have verified that both client and server-side (using Munin-Tsung integration) profiling data are able to be aggregated by Tsung into an HTML format with a single command. It should be possible to simply host a web server on the driver machine to access these reports immediately after execution. JMeter has integration with Jenkins, which allows it to pull performance testing data from JMeter. I am confident that neither tool will post a barrier to being able to run tests nightly. >> 4. Is the level of quality of the test results acceptable? I don't >> have any doubts that both frameworks offer sufficient data, but I >> would like to see which information is available once you've run it >> through the recommended automation process All of JMeter's and Tsung's data is publishable VIA their automated approaches. >> 5. License? Given this is a support tool, as long as the license >> allows free use, I think we're ok. right? All good for both. In conclusion, I feel that the Tsung-based Open-Performance-Automation-Framework is the better option moving forward. While most aspects remain quite equal, I feel as though the Tsung tests will be more maintainable and re-usable in the long term. Thanks to everyone for your valuable input and effort into this. It certainly helped the process along. Cheers, Branden [1] https://github.com/AlanBerg/SakaiOAE-Open/tree/master/TESTS/jmeter_performance_01 [2] https://github.com/mrvisser/Open-Performance-Automation-Framework/tree/oae-load-tests On Thu, Jul 19, 2012 at 11:17 AM, Kyle Campos <[email protected]> wrote: > On Wed, Jul 18, 2012 at 11:48 AM, Branden Visser <[email protected]> wrote: >> >> [snip] >> >> >> As for selection criteria for a decision, I hope to learn the >> following from the frameworks in this trial (in no particular order of >> priority): >> >> 1. How easy is it to create a re-usable test case. Is it feasible to >> think that an institution will pull the components together needed to >> contribute back to the effort? > > > I went a different direction with Tsung. I wanted a re-usable API that > institutions/implementers could use to quickly write their own tests for > their own use cases. Knowing that there's no way a performance test > "artifact" (in Tsun'gs case the output xml) would be executable or valuable > OOTB in all those differences deployment environments with all their > different customizations. This method has proven true and reliable in many > cases. We've used it for multiple Kuali products across multiple > institutions/implementations. > >> >> 2. Which is more feasible for the *test maintainers* to keep updated? >> I highlight test maintainers, because at the end of the day, if those >> who will ultimately be responsible for maintaining the tests are not >> comfortable with the medium, I think the effort will fail > > > I've expressed my opinion here previously. > >> >> 3. What effort is required to perform the tests and get the data into >> a valuable format -- is it feasible to do weekly? nightly? > > > Tsung framework launched through the cmd line. So automating that is a 1 > line sh command in Jenkins. > >> >> 4. Is the level of quality of the test results acceptable? I don't >> have any doubts that both frameworks offer sufficient data, but I >> would like to see which information is available once you've run it >> through the recommended automation process > > > There's 2 parts to the results, the client side HTTP statistics Tsung would > gather and then the server side resource utilization results. We can parse > and present the HTTP statistics any way we want, but OOTB Tsung has a script > that turns it into HTML. > > On the server side, I created a monitoring tool[1] that synchronizes with > the Tsung test and fires off monitoring "listeners" across an array of > servers asynchronously. It uses all free unix utilities to gather > utilization data (cpu, mem etc...). It currently uses gruff graphs to chart > those results for some data types, but I'd like to change this to using > Google Charts for all the data it gathers. That's not a huge effort and one > I'd actually be excited to do were you interested in using. > > In short I have both sides of the equation there and ready to use in an > automated way. > >> 5. License? Given this is a support tool, as long as the license >> allows free use, I think we're ok. right? > > > Both of mine are ECL. > >> >> Am I missing any important points? > > > Not that I can think of. Let me know if I can be of any assistance. > > -Kyle > > [1] https://github.com/kcampos/Open-Synchronized-System-Resource-Monitoring > >> >> Thanks, >> Branden >> >> On Wed, Jul 18, 2012 at 12:48 PM, Lance Speelmon <[email protected]> wrote: >> > This is all really great discussion. How are we going to drive this to >> > conclusion? We likely are going to need to start writing tests using >> > this >> > tool in the next week or two as part of the first 1.5.x sprint. >> > Suggestions: >> > >> > Adopt an appropriate decision time box - i.e. one or two weeks. >> > Identify selection criteria. Maybe using an existing set of criteria? >> > i.e.: >> > >> > ease of use for developers (APIs, etc.) >> > ease of use for deployers (e.g. running load tests against localized >> > builds) >> > strength of the community >> > suitable license >> > proven track record (success stories in applications somewhat like ours) >> > options for queries >> > options for scaling >> > >> > Examine outcomes >> > Make a decision >> > >> > >> > WDYT? Thanks, L >> > >> > >> > On Jul 18, 2012, at 5:48 AM, Juan Jose Meroño Sanchez <[email protected]> >> > wrote: >> > >> > Hi All, >> > >> > I'm Juan, Alan asked me to join this conversation to give my point >> > of >> > view, thanks !! >> > >> > I think the real important thing is the content not the tool, because >> > all >> > the >> > tools you are evaluating are really good (jmeter, tsung, grinder,..). >> > >> > The framework that I wrote is focus on reduce the manual tasks needed to >> > run >> > performance tests, >> > and try to automatize as much as possible. This allows to add >> > performance >> > tests to software lifecycle >> > and keep performance always in mind, not only when server goes down :-) >> > >> > Also anyone can download and easyly run tests with his own Sakai >> > distribution (without any knowledge of the tool). >> > >> > Anyway, you can achieve this goal with any tool. In my case I picked up >> > JMeter because: >> > >> > 1.- That's the tool I knew. >> > 2.- There are maven plugins ready to execute JMeter tests. >> > >> > If community testers feel more confortable with tsung, or other tools, >> > you >> > only need a maven plugin or >> > ant task that allows to run it. >> > Because of the tests cases are the really valuable thing, the best test >> > tool >> > is the one with more tests :-) >> > >> > Bye, Juan. >> > >> > El 18/07/2012 9:26, Berg, Alan escribió: >> > >> > Hi Kyle, >> > >> > Thanks for taking the time and writing well though out comments. >> > I should take the time to understand tsung in detail, it sounds like a >> > viable solution. >> > >> > I agree with you on: >> > >> >>At their core, both jmeter and tsung are great performance tools, the >> >> question is how do you want to work with the performance framework >> >> inside >> >> the OAE dev community and with implementers at large. That should guide >> >> selection criteria IMO. >> > >> >> If it's a requirement that the community delivery a framework that >> >> implementers can use to performance test their implementations, I'd >> >> love to >> >> hear the argument for jmeter in that case. >> > >> > One partial solution: Write a data driven test plan for Jmeter which has >> > the >> > location of all API's in a text file and make it run through the full >> > API. >> > Have other text files for the input to the API. You could do this as a >> > kind >> > of Performance unit test. If you add assertions for response time or >> > some >> > such, you can add the thresholds to the same text files. Connect that up >> > to >> > Jenkins and make the updating of the data part of the development >> > process. >> > >> > The other part of the solution is to define some standard criteria which >> > can >> > be re-used across performance tools. For example, sizes for provisioned >> > type: smalll, medium, large and try and develop an understanding of the >> > generic mix between. This was done in the Framework for CLE. Which I >> > would >> > argue we need to extend and simplify (perhaps) for OAE. The data mix >> > definitions can be reused across performance tools. >> > >> > >From the JavaScript side you would need something like selenium >> > webdriver >> > to trigger Qunit tests. >> > >> > Sorry cant resist a plug for Jmeter in this election year: The example >> > code >> > I mentioned runs from maven with no setup cost (mvn verify) and >> > generates a >> > report. You can just hook that into the Jmeter plugin in Jenkins which >> > then >> > consumes the results. To add an extra test plan to run you would need to >> > dump it into the /src/test/jmeter directory. The test plans are >> > examples, >> > which need expanding. Juans work looks very interesting, but if there is >> > a >> > setup cost then more work would need to be done to lower that cost. >> > >> > I would say try it out and give feedback. >> > >> > >> > https://github.com/AlanBerg/SakaiOAE-Open/tree/master/TESTS/jmeter_microbenchmark >> > >> > >> > It does assume that you have a demo running on port 8080. However, to >> > change >> > this look at /src/test/jmeter/user.properties and tweak the settings. >> > >> > >> > Thanks Kyle for your response. >> > >> > >> > Regards, >> > >> > >> > Alan >> > >> > >> > >> > Alan Berg >> > >> > Group Education and Research Services >> > Central Computer Services >> > University of Amsterdam >> > ________________________________ >> > From: Kyle Campos [[email protected]] >> > Sent: 17 July 2012 20:19 >> > To: Berg, Alan >> > Cc: Branden Visser; [email protected] >> > >> > Subject: Re: [oae-dev] Load testing tool >> > >> > Hi, >> > >> > I want to make sure I'm commenting on items that actually mean something >> > to >> > you guys, so I'd love to have someone tell me what the requirements are >> > for >> > the performance framework and what selection criteria is in place. I >> > only >> > know of one, that it be open source. >> > >> > I'll respond inline below to some of your points, but the biggest point >> > for >> > me is the point I made about what gets delivered to implementers. If >> > it's a >> > requirement that the community delivery a framework that implementers >> > can >> > use to performance test their implementations, I'd love to hear the >> > argument >> > for jmeter in that case. As I mentioned, I don't see jmeter viable in >> > that >> > scenario. If this isn't a requirement and implementers need to roll >> > their >> > own solution or test a OOTB community deployment then jmeter would be >> > fine. >> > If it is a requirement, the only way I see you being able to meet it is >> > to >> > support an API that implementers can use to develop their use cases. The >> > community won't know what features implementers have >> > enabled/disabled/customized, they won't know the deployment >> > architecture. >> > It's nearly impossible to deliver a static set of tests from the >> > community >> > to all implementers and say, "go click run and all should work". >> > >> > Rest of comments inline below... >> > >> > On Tue, Jul 17, 2012 at 1:27 AM, Berg, Alan <[email protected]> wrote: >> >> >> >> Hi fellow hard workers, >> >> >> >> Good to have this discussion, I can learn from my peers. I will argue >> >> for >> >> Jmeter below. I am certain it is a viable tool, however, I am not >> >> saying >> >> that tsung is not also a viable tool. I want to make a fair comparison, >> >> so >> >> need to note some differences in emphasis. >> >> >> >> In terms of maintainability, it is important for stress tests to be as >> >> simple as is possible and data driven. It does not matter which >> >> technology >> >> you use if you don't follow conventions and basic design patterns. >> >> >> >> >> >> > I much prefer handling this programatically, >> >> >> >> I found it straightforward to mentor a functional administrator to >> >> stress >> >> test using Jmeter. The GUI to create tests with its reverse proxy is >> >> not >> >> difficult to explain. True: The plans are saved in a more complex than >> >> Tsung >> >> XML format.In terms of recording tests i tend to use badboy >> >> (http://www.badboy.com.au/), save in Jmeter format and tweak. Give it a >> >> go. >> > >> > >> > Couple points here: >> > >> > 1. I understad that showing someone the GUI seems to make things easier, >> > and >> > it may for some, especially non-technical people (BA, SME types). But I >> > find >> > the GUI gets in the way of rapid test developmen. >> > >> > 2. I never understood the value of getting non-technical people running >> > performance tests. Even if the GUI makes generating a performance test >> > easy, >> > there's the setup of the system and resource monitoring, then collecting >> > all >> > the data and making sense of it. Performance tests aren't functional >> > tests, >> > whether a performance test passes or fails, is good or bad, requires a >> > good >> > amount of technical expertise. >> > >> > So the argument here really boils down to GUI vs. API. I'd just >> > encourage >> > you to evaluate all the implications of the GUI workflow. >> > >> >> >> >> Jmeter has lots of assertions including reglex. You simply add the >> >> assertions as children under the http samplers. >> >> >> >> >> >> > Dynamic Variables - again it's jmeter's Bean Shell PreProcessor >> >> > workflow >> >> > vs tsung's regexp param in the http request XML >> >> >> >> You can use dynamic variables in Jmeter as well. Here are the list of >> >> functions to use in any sampler and the definition of variables. >> >> http://jmeter.apache.org/usermanual/functions.html >> >> >> >> If all this wiring is not enough, then you can fall back to the >> >> beanshell. >> >> At this point you are moving away from KISS and that should be a >> >> warning >> >> about maintainability. >> > >> > >> > When I use the term "dynamic variables" I'm referring to variables that >> > are >> > only set at runtime, are thread specific and are set by parsing some >> > previous requests HTTP response. AFAIK that type of variable must be set >> > via >> > jmeter's beanshell preprocessor. >> > >> > In any case the point is, its not as straight forward as tsung. >> > >> >> There is a working example of a framework of Jmeter in CLE land which >> >> can >> >> work with CLE, Hybrid and extended to OAE. This will allow us to share >> >> data >> >> models across communities. If later some one wishes to move to a hybrid >> >> instance then they can leverage there knowledge from CLE land. Now, it >> >> is >> >> true that this is currently not a reality (as Lance fairly pointed >> >> out), >> >> however, if we plough the land then seeds can grow. >> >> >> >> There are plenty of examples of Jmeter used at large scale with a large >> >> number of developers. It has a well established community. >> >> >> >> Here is a book on the subject: >> >> http://www.packtpub.com/beginning-apache-jmeter/book >> >> >> >> Here are some links: >> >> http://wiki.apache.org/jmeter/JMeterLinks/ >> >> >> >> Here is a cloud service: >> >> http://blazemeter.com/ >> > >> > >> > I understand the community is large, really large actually. But the way >> > you >> > are forced to collaborate, or the limitations of collaboration on the >> > actual >> > framework that's developed is not optimal mostly due to the GUI >> > interface. >> > We'll all be forced to re-record use cases as implementers, so I'm not >> > sure >> > what value implementers would be leveraging from any community work. >> > Were >> > the community to release a tagged API, implementers would just write >> > very >> > light weight test cases that exercise the API. But again, that goes back >> > to >> > requirements and selection criteria. >> > >> > At their core, both jmeter and tsung are great performance tools, the >> > question is how do you want to work with the performance framework >> > inside >> > the OAE dev community and with implementers at large. That should guide >> > selection criteria IMO. >> > >> > Thanks Alan >> > -Kyle >> > >> >> Jmeters main weakness is that it does not understand JavaScript easily. >> >> Selenium webdriver with Qunit is the way forward for that. >> >> >> >> Looking forward to a detailed response. >> >> >> >> Alan >> >> >> >> >> >> >> >> >> >> >> >> Alan Berg >> >> >> >> Group Education and Research Services >> >> Central Computer Services >> >> University of Amsterdam >> >> ________________________________ >> >> From: [email protected] >> >> [[email protected]] on behalf of Kyle Campos >> >> [[email protected]] >> >> Sent: 17 July 2012 03:25 >> >> >> >> To: Branden Visser >> >> Cc: [email protected] >> >> Subject: Re: [oae-dev] Load testing tool >> >> >> >> Branden, >> >> >> >> I'll just jump into the technical reasons I see jmeter being more >> >> difficult to work with from a community perspective. Really you touched >> >> on >> >> it in your positive point #2 for Tsung and that's Tsung's XML >> >> structure. But >> >> the implications of this deserve more highlight especially in the >> >> context of >> >> the community requirement for automated/nightly performance tests. >> >> >> >> What I don't like about jmeter is primarily its GUI dependency and the >> >> implications of it on workflow, extensibility, collaboration etc... It >> >> makes >> >> for really slow test development with large use cases and more >> >> difficult >> >> maintenance/collaboration between teams. Concrete examples below... >> >> >> >> 1. AJAX request handling - I don't think they could have made it >> >> any >> >> more convoluted with their "logic controller". Using logic "wizards" in >> >> a >> >> GUI is just ugly and makes me cringe. I much prefer handling this >> >> programatically, which is what I did in tsung and what our abstraction >> >> layer >> >> makes really easy and transparent to test writers. If you want the >> >> pain, >> >> then go through jmeter's GUI workflow for developing the logic around >> >> username lookup on signup, then factor in dynamic substitution with >> >> reading >> >> in usernames from an external file, now think about how easy it would >> >> be for >> >> someone else to change this logic at runtime. Ick. >> >> >> >> There's 1 place that controls this in my tsung framework. You don't >> >> need to go through this pain at all writing test cases, and even if you >> >> built it from scratch it's very simple. >> >> >> >> 2. Dynamic Variables - again it's jmeter's Bean Shell PreProcessor >> >> workflow vs tsung's regexp param in the http request XML. And again >> >> this is >> >> also wrapped in our framework. >> >> >> >> Both of the above examples will be in heavy use in any good OAE >> >> performance test. >> >> >> >> I think your point about releasing performance tests as an artifact >> >> with >> >> the release is good in principle, but I don't see jmeter being the best >> >> vehicle to deliver those. As a deployer myself I'd much rather the >> >> community >> >> provide a tagged API set that I can leverage to build that profile MY >> >> use >> >> cases(our tsung framework is built with that in mind). I don't want a >> >> set of >> >> static scripts that may or may not execute in my environment and that >> >> may or >> >> may not profile anything of use for my implementation. There's no way >> >> the >> >> community can know those things or build performance tests that address >> >> all >> >> those use cases. >> >> >> >> I've gone through this technology selection with a more broad set of >> >> requirements than most folks who use jmeter need. jmeter is a very >> >> common >> >> developer tool to quickly script up a simple performance test. I've >> >> never >> >> seen it used very successfully in a broad context with many devs >> >> contributing, with it running complex use cases, against a young code >> >> base >> >> and it being maintainable over time. That being said, you are all very >> >> talented and I'm sure you could get it to work for you, but I'd be very >> >> careful that you don't paint yourself into a corner with burdensome >> >> maintenance and test development workflow that limits contribution. >> >> >> >> My $0.02 >> >> >> >> -Kyle >> >> >> >> On Mon, Jul 16, 2012 at 4:42 AM, Branden Visser <[email protected]> >> >> wrote: >> >>> >> >>> Hi everyone, >> >>> >> >>> I've been putting some research into an appropriate load testing tool >> >>> for us to use, I have been focusing mainly on JMeter and Tsung. >> >>> >> >>> Tsung, as far as I can tell, has the following selling points: >> >>> >> >>> 1. Its erlang guts let it drive many concurrent users more efficiently >> >>> than its competitors >> >>> 2. Its XML structure seems quite leaner, making it easier build tests >> >>> from source >> >>> 3. There is existing work from rSmart that can be leveraged to drive >> >>> our performance tests [1] >> >>> >> >>> With JMeter, I see the following benefits: >> >>> >> >>> 1. Lower overhead in spinning up a test, once the tests are already >> >>> built (VIA a maven plugin) >> >>> 2. I think the increase in complexity of JMeter comes with the benefit >> >>> of extensibility (unless you know erlang, I guess..) >> >>> 3. There is an existing community effort that can be leveraged to >> >>> drive our performance tests [2] >> >>> >> >>> They both have exactly 3 advantages, I don't know what to do!? >> >>> >> >>> Just kidding. But, unless there is quantifiable evidence that suggests >> >>> JMeter's performance will not suffice to properly test OAE (I have not >> >>> been able to find such evidence yet, but others may have more data), I >> >>> propose that we move forward with JMeter. I see value in making the >> >>> JMeter tests executable from the same command-line on which OAE is >> >>> built. I think this moves towards making the JMeter tests an artifact >> >>> of the release and not some orthogonal set of scripts uploaded >> >>> elsewhere, which in my experience tend to become of questionable age >> >>> and relevance. I think it will become more valuable to our deployers, >> >>> and the deployers' performance test data (which would hopefully be >> >>> more abundant with the lower barrier of entry) will become more >> >>> valuable to the core team. >> >>> >> >>> [1] https://github.com/kcampos/Open-Performance-Automation-Framework >> >>> [2] >> >>> https://confluence.sakaiproject.org/display/QA/CLE+Load+Test+Framework >> >>> >> >>> -- >> >>> Cheers, >> >>> Branden >> >>> _______________________________________________ >> >>> oae-dev mailing list >> >>> [email protected] >> >>> http://collab.sakaiproject.org/mailman/listinfo/oae-dev >> >> >> >> >> >> >> >> >> >> -- >> >> Kyle Campos >> >> Director of Quality Operations / rSmart >> >> [email protected] >> >> skype: kyle.campos >> >> phone: 623-455-6180 >> >> GTalk: [email protected] >> >> >> > >> > >> > >> > -- >> > Kyle Campos >> > Director of Quality Operations / rSmart >> > [email protected] >> > skype: kyle.campos >> > phone: 623-455-6180 >> > GTalk: [email protected] >> > >> > >> > >> > _______________________________________________ >> > oae-dev mailing list >> > [email protected] >> > http://collab.sakaiproject.org/mailman/listinfo/oae-dev >> > >> > >> > _______________________________________________ >> > oae-dev mailing list >> > [email protected] >> > http://collab.sakaiproject.org/mailman/listinfo/oae-dev >> > >> > >> > >> > _______________________________________________ >> > oae-dev mailing list >> > [email protected] >> > http://collab.sakaiproject.org/mailman/listinfo/oae-dev >> > >> _______________________________________________ >> oae-dev mailing list >> [email protected] >> http://collab.sakaiproject.org/mailman/listinfo/oae-dev > > > > > -- > Kyle Campos > Director of Quality Operations / rSmart > [email protected] > skype: kyle.campos > phone: 623-455-6180 > GTalk: [email protected] > _______________________________________________ oae-dev mailing list [email protected] http://collab.sakaiproject.org/mailman/listinfo/oae-dev _______________________________________________ oae-dev mailing list [email protected] http://collab.sakaiproject.org/mailman/listinfo/oae-dev
