I'll try to answer your questions:
> 1. How easy is it to create a re-usable test case. Is it feasible to
> think that an institution will pull the components together needed to
> contribute back to the effort?
I think this is the main problem. For example, if you are using asserts
you need to check
response content avoiding localized strings (you should use html tags or
style classes) to
detect bad responses (of course you should check http response code too).
Also you need to create regular expressions to navigate within the
portal (how to find the next link?)
In the other hand you need to populate the environment. For example, If
you create a case to test
Sakai 2 sites inside OAE, you have to ensure that test users have access
to any Sakai 2 site in the test server.
How to get environment ready before run tests?
Giving manual instructions to the tester?
Creating a big scenario (even a copy of a real situation) and using
it on every test?
Create a webservices test case to populate the environment?
I think best could be a combination of last two questions. Maybe we can
create a webservices test case to
create a big scenario (and other to drop it), and run this test monthly.
All the other tests will run based on this
big scenario.
> 2. Which is more feasible for the *test maintainers* to keep updated?
> I highlight test maintainers, because at the end of the day, if those
> who will ultimately be responsible for maintaining the tests are not
> comfortable with the medium, I think the effort will fail
Test cases are sensitive to changes on url/link formats, strings used in
asserts, and environment data.
So, if you create a good test, probably works in every version of sakai
until url format or page composition change.
In cle test framework test cases version are independent from test
framework version, so you can choose the version
of test case you want to run.
> 3. What effort is required to perform the tests and get the data into
> a valuable format -- is it feasible to do weekly? nightly?
It depends on the resources available for the server tested. In JMeter
you could run a test during a fixed period of time,
or run the test until the end of loops. So you can run test nightly (for
low stress tests, less than two hours duration), weekly (for high stress
tests, more than two hours) !!
> 4. Is the level of quality of the test results acceptable? I don't
> have any doubts that both frameworks offer sufficient data, but I
> would like to see which information is available once you've run it
> through the recommended automation process
With JMeter maven plugin you obtain something like this:
https://confluence.sakaiproject.org/download/attachments/75666456/CreateResults.jpg?version=1&modificationDate=1316510899000
<https://confluence.sakaiproject.org/download/attachments/75666456/CreateResults.jpg?version=1&modificationDate=1316510899000>
Raw table with average response time for each request.
Graph with response time and throughput with request/second and thread
count.
I think is enough for client side, is useful collect server performance
data too, but automatize this is quite difficult.
> 5. License? Given this is a support tool, as long as the license
> allows free use, I think we're ok. right?
CLE Test framework mainly uses:
JMeter: http://www.apache.org/licenses/
Maven Chronos Plugin: http://mojo.codehaus.org/chronos/license.html
> Am I missing any important points?
Important points reviewed !!
Bye, Juan.
El 18/07/2012 20:48, Branden Visser escribió:
> Agree, but in the interest of time I would like to keep the choice
> narrowed down to Tsung and JMeter, and more specifically the
> respective frameworks that are currently under development within the
> community. They were both designed with a long-term goal to allow
> deployers to reuse work within the community, and I think we should
> try and stick to that.
>
> Also, I would like to ensure we achieve the short-term goal of getting
> valuable performance data out of OAE to help drive the 1.5.x sprints.
> I think the first step is to get the same coverage out of our
> performance testing scripts as we have with the current NeoLoad tests:
>
> General Search
> InviteContact
> AcceptContact
> View Library
> Update profile
> View Messages
> View Memberships
> View Contacts
>
> Alan has agreed to script these tests on the JMeter framework and
> integrate the results with Jenkins as a proof of concept, and
> hopefully that can be demonstrated Friday. In the meantime, I have
> started working to prepare the same cases with the Tsung-based
> framework with some help from Kyle -- any guidance on how we can
> automate the production of test results in this area is appreciated,
> if you have expertise on this, please let me know! Also, if you'd like
> to jump in and help prepare some Tsung test cases for the above, I'm
> all open to volunteers :)
>
> As for selection criteria for a decision, I hope to learn the
> following from the frameworks in this trial (in no particular order of
> priority):
>
> 1. How easy is it to create a re-usable test case. Is it feasible to
> think that an institution will pull the components together needed to
> contribute back to the effort?
> 2. Which is more feasible for the *test maintainers* to keep updated?
> I highlight test maintainers, because at the end of the day, if those
> who will ultimately be responsible for maintaining the tests are not
> comfortable with the medium, I think the effort will fail
> 3. What effort is required to perform the tests and get the data into
> a valuable format -- is it feasible to do weekly? nightly?
> 4. Is the level of quality of the test results acceptable? I don't
> have any doubts that both frameworks offer sufficient data, but I
> would like to see which information is available once you've run it
> through the recommended automation process
> 5. License? Given this is a support tool, as long as the license
> allows free use, I think we're ok. right?
>
> Am I missing any important points?
>
> Thanks,
> Branden
>
> On Wed, Jul 18, 2012 at 12:48 PM, Lance Speelmon<[email protected]> wrote:
>> This is all really great discussion. How are we going to drive this to
>> conclusion? We likely are going to need to start writing tests using this
>> tool in the next week or two as part of the first 1.5.x sprint.
>> Suggestions:
>>
>> Adopt an appropriate decision time box - i.e. one or two weeks.
>> Identify selection criteria. Maybe using an existing set of criteria? i.e.:
>>
>> ease of use for developers (APIs, etc.)
>> ease of use for deployers (e.g. running load tests against localized builds)
>> strength of the community
>> suitable license
>> proven track record (success stories in applications somewhat like ours)
>> options for queries
>> options for scaling
>>
>> Examine outcomes
>> Make a decision
>>
>>
>> WDYT? Thanks, L
>>
>>
>> On Jul 18, 2012, at 5:48 AM, Juan Jose Meroño Sanchez<[email protected]>
>> wrote:
>>
>> Hi All,
>>
>> I'm Juan, Alan asked me to join this conversation to give my point of
>> view, thanks !!
>>
>> I think the real important thing is the content not the tool, because all
>> the
>> tools you are evaluating are really good (jmeter, tsung, grinder,..).
>>
>> The framework that I wrote is focus on reduce the manual tasks needed to run
>> performance tests,
>> and try to automatize as much as possible. This allows to add performance
>> tests to software lifecycle
>> and keep performance always in mind, not only when server goes down :-)
>>
>> Also anyone can download and easyly run tests with his own Sakai
>> distribution (without any knowledge of the tool).
>>
>> Anyway, you can achieve this goal with any tool. In my case I picked up
>> JMeter because:
>>
>> 1.- That's the tool I knew.
>> 2.- There are maven plugins ready to execute JMeter tests.
>>
>> If community testers feel more confortable with tsung, or other tools, you
>> only need a maven plugin or
>> ant task that allows to run it.
>> Because of the tests cases are the really valuable thing, the best test tool
>> is the one with more tests :-)
>>
>> Bye, Juan.
>>
>> El 18/07/2012 9:26, Berg, Alan escribió:
>>
>> Hi Kyle,
>>
>> Thanks for taking the time and writing well though out comments.
>> I should take the time to understand tsung in detail, it sounds like a
>> viable solution.
>>
>> I agree with you on:
>>
>>> At their core, both jmeter and tsung are great performance tools, the
>>> question is how do you want to work with the performance framework inside
>>> the OAE dev community and with implementers at large. That should guide
>>> selection criteria IMO.
>>> If it's a requirement that the community delivery a framework that
>>> implementers can use to performance test their implementations, I'd love to
>>> hear the argument for jmeter in that case.
>> One partial solution: Write a data driven test plan for Jmeter which has the
>> location of all API's in a text file and make it run through the full API.
>> Have other text files for the input to the API. You could do this as a kind
>> of Performance unit test. If you add assertions for response time or some
>> such, you can add the thresholds to the same text files. Connect that up to
>> Jenkins and make the updating of the data part of the development process.
>>
>> The other part of the solution is to define some standard criteria which can
>> be re-used across performance tools. For example, sizes for provisioned
>> type: smalll, medium, large and try and develop an understanding of the
>> generic mix between. This was done in the Framework for CLE. Which I would
>> argue we need to extend and simplify (perhaps) for OAE. The data mix
>> definitions can be reused across performance tools.
>>
>> > From the JavaScript side you would need something like selenium webdriver
>> to trigger Qunit tests.
>>
>> Sorry cant resist a plug for Jmeter in this election year: The example code
>> I mentioned runs from maven with no setup cost (mvn verify) and generates a
>> report. You can just hook that into the Jmeter plugin in Jenkins which then
>> consumes the results. To add an extra test plan to run you would need to
>> dump it into the /src/test/jmeter directory. The test plans are examples,
>> which need expanding. Juans work looks very interesting, but if there is a
>> setup cost then more work would need to be done to lower that cost.
>>
>> I would say try it out and give feedback.
>>
>> https://github.com/AlanBerg/SakaiOAE-Open/tree/master/TESTS/jmeter_microbenchmark
>>
>>
>> It does assume that you have a demo running on port 8080. However, to change
>> this look at /src/test/jmeter/user.properties and tweak the settings.
>>
>>
>> Thanks Kyle for your response.
>>
>>
>> Regards,
>>
>>
>> Alan
>>
>>
>>
>> Alan Berg
>>
>> Group Education and Research Services
>> Central Computer Services
>> University of Amsterdam
>> ________________________________
>> From: Kyle Campos [[email protected]]
>> Sent: 17 July 2012 20:19
>> To: Berg, Alan
>> Cc: Branden Visser; [email protected]
>>
>> Subject: Re: [oae-dev] Load testing tool
>>
>> Hi,
>>
>> I want to make sure I'm commenting on items that actually mean something to
>> you guys, so I'd love to have someone tell me what the requirements are for
>> the performance framework and what selection criteria is in place. I only
>> know of one, that it be open source.
>>
>> I'll respond inline below to some of your points, but the biggest point for
>> me is the point I made about what gets delivered to implementers. If it's a
>> requirement that the community delivery a framework that implementers can
>> use to performance test their implementations, I'd love to hear the argument
>> for jmeter in that case. As I mentioned, I don't see jmeter viable in that
>> scenario. If this isn't a requirement and implementers need to roll their
>> own solution or test a OOTB community deployment then jmeter would be fine.
>> If it is a requirement, the only way I see you being able to meet it is to
>> support an API that implementers can use to develop their use cases. The
>> community won't know what features implementers have
>> enabled/disabled/customized, they won't know the deployment architecture.
>> It's nearly impossible to deliver a static set of tests from the community
>> to all implementers and say, "go click run and all should work".
>>
>> Rest of comments inline below...
>>
>> On Tue, Jul 17, 2012 at 1:27 AM, Berg, Alan<[email protected]> wrote:
>>> Hi fellow hard workers,
>>>
>>> Good to have this discussion, I can learn from my peers. I will argue for
>>> Jmeter below. I am certain it is a viable tool, however, I am not saying
>>> that tsung is not also a viable tool. I want to make a fair comparison, so
>>> need to note some differences in emphasis.
>>>
>>> In terms of maintainability, it is important for stress tests to be as
>>> simple as is possible and data driven. It does not matter which technology
>>> you use if you don't follow conventions and basic design patterns.
>>>
>>>
>>>> I much prefer handling this programatically,
>>> I found it straightforward to mentor a functional administrator to stress
>>> test using Jmeter. The GUI to create tests with its reverse proxy is not
>>> difficult to explain. True: The plans are saved in a more complex than Tsung
>>> XML format.In terms of recording tests i tend to use badboy
>>> (http://www.badboy.com.au/), save in Jmeter format and tweak. Give it a go.
>>
>> Couple points here:
>>
>> 1. I understad that showing someone the GUI seems to make things easier, and
>> it may for some, especially non-technical people (BA, SME types). But I find
>> the GUI gets in the way of rapid test developmen.
>>
>> 2. I never understood the value of getting non-technical people running
>> performance tests. Even if the GUI makes generating a performance test easy,
>> there's the setup of the system and resource monitoring, then collecting all
>> the data and making sense of it. Performance tests aren't functional tests,
>> whether a performance test passes or fails, is good or bad, requires a good
>> amount of technical expertise.
>>
>> So the argument here really boils down to GUI vs. API. I'd just encourage
>> you to evaluate all the implications of the GUI workflow.
>>
>>> Jmeter has lots of assertions including reglex. You simply add the
>>> assertions as children under the http samplers.
>>>
>>>
>>>> Dynamic Variables - again it's jmeter's Bean Shell PreProcessor workflow
>>>> vs tsung's regexp param in the http request XML
>>> You can use dynamic variables in Jmeter as well. Here are the list of
>>> functions to use in any sampler and the definition of variables.
>>> http://jmeter.apache.org/usermanual/functions.html
>>>
>>> If all this wiring is not enough, then you can fall back to the beanshell.
>>> At this point you are moving away from KISS and that should be a warning
>>> about maintainability.
>>
>> When I use the term "dynamic variables" I'm referring to variables that are
>> only set at runtime, are thread specific and are set by parsing some
>> previous requests HTTP response. AFAIK that type of variable must be set via
>> jmeter's beanshell preprocessor.
>>
>> In any case the point is, its not as straight forward as tsung.
>>
>>> There is a working example of a framework of Jmeter in CLE land which can
>>> work with CLE, Hybrid and extended to OAE. This will allow us to share data
>>> models across communities. If later some one wishes to move to a hybrid
>>> instance then they can leverage there knowledge from CLE land. Now, it is
>>> true that this is currently not a reality (as Lance fairly pointed out),
>>> however, if we plough the land then seeds can grow.
>>>
>>> There are plenty of examples of Jmeter used at large scale with a large
>>> number of developers. It has a well established community.
>>>
>>> Here is a book on the subject:
>>> http://www.packtpub.com/beginning-apache-jmeter/book
>>>
>>> Here are some links:
>>> http://wiki.apache.org/jmeter/JMeterLinks/
>>>
>>> Here is a cloud service:
>>> http://blazemeter.com/
>>
>> I understand the community is large, really large actually. But the way you
>> are forced to collaborate, or the limitations of collaboration on the actual
>> framework that's developed is not optimal mostly due to the GUI interface.
>> We'll all be forced to re-record use cases as implementers, so I'm not sure
>> what value implementers would be leveraging from any community work. Were
>> the community to release a tagged API, implementers would just write very
>> light weight test cases that exercise the API. But again, that goes back to
>> requirements and selection criteria.
>>
>> At their core, both jmeter and tsung are great performance tools, the
>> question is how do you want to work with the performance framework inside
>> the OAE dev community and with implementers at large. That should guide
>> selection criteria IMO.
>>
>> Thanks Alan
>> -Kyle
>>
>>> Jmeters main weakness is that it does not understand JavaScript easily.
>>> Selenium webdriver with Qunit is the way forward for that.
>>>
>>> Looking forward to a detailed response.
>>>
>>> Alan
>>>
>>>
>>>
>>>
>>>
>>> Alan Berg
>>>
>>> Group Education and Research Services
>>> Central Computer Services
>>> University of Amsterdam
>>> ________________________________
>>> From: [email protected]
>>> [[email protected]] on behalf of Kyle Campos
>>> [[email protected]]
>>> Sent: 17 July 2012 03:25
>>>
>>> To: Branden Visser
>>> Cc: [email protected]
>>> Subject: Re: [oae-dev] Load testing tool
>>>
>>> Branden,
>>>
>>> I'll just jump into the technical reasons I see jmeter being more
>>> difficult to work with from a community perspective. Really you touched on
>>> it in your positive point #2 for Tsung and that's Tsung's XML structure. But
>>> the implications of this deserve more highlight especially in the context of
>>> the community requirement for automated/nightly performance tests.
>>>
>>> What I don't like about jmeter is primarily its GUI dependency and the
>>> implications of it on workflow, extensibility, collaboration etc... It makes
>>> for really slow test development with large use cases and more difficult
>>> maintenance/collaboration between teams. Concrete examples below...
>>>
>>> 1. AJAX request handling - I don't think they could have made it any
>>> more convoluted with their "logic controller". Using logic "wizards" in a
>>> GUI is just ugly and makes me cringe. I much prefer handling this
>>> programatically, which is what I did in tsung and what our abstraction layer
>>> makes really easy and transparent to test writers. If you want the pain,
>>> then go through jmeter's GUI workflow for developing the logic around
>>> username lookup on signup, then factor in dynamic substitution with reading
>>> in usernames from an external file, now think about how easy it would be for
>>> someone else to change this logic at runtime. Ick.
>>>
>>> There's 1 place that controls this in my tsung framework. You don't
>>> need to go through this pain at all writing test cases, and even if you
>>> built it from scratch it's very simple.
>>>
>>> 2. Dynamic Variables - again it's jmeter's Bean Shell PreProcessor
>>> workflow vs tsung's regexp param in the http request XML. And again this is
>>> also wrapped in our framework.
>>>
>>> Both of the above examples will be in heavy use in any good OAE
>>> performance test.
>>>
>>> I think your point about releasing performance tests as an artifact with
>>> the release is good in principle, but I don't see jmeter being the best
>>> vehicle to deliver those. As a deployer myself I'd much rather the community
>>> provide a tagged API set that I can leverage to build that profile MY use
>>> cases(our tsung framework is built with that in mind). I don't want a set of
>>> static scripts that may or may not execute in my environment and that may or
>>> may not profile anything of use for my implementation. There's no way the
>>> community can know those things or build performance tests that address all
>>> those use cases.
>>>
>>> I've gone through this technology selection with a more broad set of
>>> requirements than most folks who use jmeter need. jmeter is a very common
>>> developer tool to quickly script up a simple performance test. I've never
>>> seen it used very successfully in a broad context with many devs
>>> contributing, with it running complex use cases, against a young code base
>>> and it being maintainable over time. That being said, you are all very
>>> talented and I'm sure you could get it to work for you, but I'd be very
>>> careful that you don't paint yourself into a corner with burdensome
>>> maintenance and test development workflow that limits contribution.
>>>
>>> My $0.02
>>>
>>> -Kyle
>>>
>>> On Mon, Jul 16, 2012 at 4:42 AM, Branden Visser<[email protected]>
>>> wrote:
>>>> Hi everyone,
>>>>
>>>> I've been putting some research into an appropriate load testing tool
>>>> for us to use, I have been focusing mainly on JMeter and Tsung.
>>>>
>>>> Tsung, as far as I can tell, has the following selling points:
>>>>
>>>> 1. Its erlang guts let it drive many concurrent users more efficiently
>>>> than its competitors
>>>> 2. Its XML structure seems quite leaner, making it easier build tests
>>>> from source
>>>> 3. There is existing work from rSmart that can be leveraged to drive
>>>> our performance tests [1]
>>>>
>>>> With JMeter, I see the following benefits:
>>>>
>>>> 1. Lower overhead in spinning up a test, once the tests are already
>>>> built (VIA a maven plugin)
>>>> 2. I think the increase in complexity of JMeter comes with the benefit
>>>> of extensibility (unless you know erlang, I guess..)
>>>> 3. There is an existing community effort that can be leveraged to
>>>> drive our performance tests [2]
>>>>
>>>> They both have exactly 3 advantages, I don't know what to do!?
>>>>
>>>> Just kidding. But, unless there is quantifiable evidence that suggests
>>>> JMeter's performance will not suffice to properly test OAE (I have not
>>>> been able to find such evidence yet, but others may have more data), I
>>>> propose that we move forward with JMeter. I see value in making the
>>>> JMeter tests executable from the same command-line on which OAE is
>>>> built. I think this moves towards making the JMeter tests an artifact
>>>> of the release and not some orthogonal set of scripts uploaded
>>>> elsewhere, which in my experience tend to become of questionable age
>>>> and relevance. I think it will become more valuable to our deployers,
>>>> and the deployers' performance test data (which would hopefully be
>>>> more abundant with the lower barrier of entry) will become more
>>>> valuable to the core team.
>>>>
>>>> [1] https://github.com/kcampos/Open-Performance-Automation-Framework
>>>> [2]
>>>> https://confluence.sakaiproject.org/display/QA/CLE+Load+Test+Framework
>>>>
>>>> --
>>>> Cheers,
>>>> Branden
>>>> _______________________________________________
>>>> oae-dev mailing list
>>>> [email protected]
>>>> http://collab.sakaiproject.org/mailman/listinfo/oae-dev
>>>
>>>
>>>
>>> --
>>> Kyle Campos
>>> Director of Quality Operations / rSmart
>>> [email protected]
>>> skype: kyle.campos
>>> phone: 623-455-6180
>>> GTalk: [email protected]
>>>
>>
>>
>> --
>> Kyle Campos
>> Director of Quality Operations / rSmart
>> [email protected]
>> skype: kyle.campos
>> phone: 623-455-6180
>> GTalk: [email protected]
>>
>>
>>
>> _______________________________________________
>> oae-dev mailing list
>> [email protected]
>> http://collab.sakaiproject.org/mailman/listinfo/oae-dev
>>
>>
>> _______________________________________________
>> oae-dev mailing list
>> [email protected]
>> http://collab.sakaiproject.org/mailman/listinfo/oae-dev
>>
>>
>>
>> _______________________________________________
>> oae-dev mailing list
>> [email protected]
>> http://collab.sakaiproject.org/mailman/listinfo/oae-dev
>>
> _______________________________________________
> oae-dev mailing list
> [email protected]
> http://collab.sakaiproject.org/mailman/listinfo/oae-dev
_______________________________________________
oae-dev mailing list
[email protected]
http://collab.sakaiproject.org/mailman/listinfo/oae-dev
