Branden,
I'll just jump into the technical reasons I see jmeter being more difficult
to work with from a community perspective. Really you touched on it in your
positive point #2 for Tsung and that's Tsung's XML structure. But the
implications of this deserve more highlight especially in the context of
the community requirement for automated/nightly performance tests.
What I don't like about jmeter is primarily its GUI dependency and the
implications of it on workflow, extensibility, collaboration etc... It
makes for really slow test development with large use cases and more
difficult maintenance/collaboration between teams. Concrete examples
below...
1. AJAX request handling - I don't think they could have made it any
more convoluted with their "logic controller". Using logic "wizards" in a
GUI is just ugly and makes me cringe. I much prefer handling this
programatically, which is what I did in tsung and what our abstraction
layer makes really easy and transparent to test writers. If you want the
pain, then go through jmeter's GUI workflow for developing the logic around
username lookup on signup, then factor in dynamic substitution with reading
in usernames from an external file, now think about how easy it would be
for someone else to change this logic at runtime. Ick.
There's 1 place that controls this in my tsung framework. You don't
need to go through this pain at all writing test cases, and even if you
built it from scratch it's very simple.
2. Dynamic Variables - again it's jmeter's Bean Shell PreProcessor
workflow vs tsung's regexp param in the http request XML. And again this is
also wrapped in our framework.
Both of the above examples will be in heavy use in any good OAE performance
test.
I think your point about releasing performance tests as an artifact with
the release is good in principle, but I don't see jmeter being the best
vehicle to deliver those. As a deployer myself I'd much rather the
community provide a tagged API set that I can leverage to build that
profile MY use cases(our tsung framework is built with that in mind). I
don't want a set of static scripts that may or may not execute in my
environment and that may or may not profile anything of use for my
implementation. There's no way the community can know those things or build
performance tests that address all those use cases.
I've gone through this technology selection with a more broad set of
requirements than most folks who use jmeter need. jmeter is a very common
developer tool to quickly script up a simple performance test. I've never
seen it used very successfully in a broad context with many devs
contributing, with it running complex use cases, against a young code base
and it being maintainable over time. That being said, you are all very
talented and I'm sure you could get it to work for you, but I'd be very
careful that you don't paint yourself into a corner with burdensome
maintenance and test development workflow that limits contribution.
My $0.02
-Kyle
On Mon, Jul 16, 2012 at 4:42 AM, Branden Visser <[email protected]> wrote:
> Hi everyone,
>
> I've been putting some research into an appropriate load testing tool
> for us to use, I have been focusing mainly on JMeter and Tsung.
>
> Tsung, as far as I can tell, has the following selling points:
>
> 1. Its erlang guts let it drive many concurrent users more efficiently
> than its competitors
> 2. Its XML structure seems quite leaner, making it easier build tests
> from source
> 3. There is existing work from rSmart that can be leveraged to drive
> our performance tests [1]
>
> With JMeter, I see the following benefits:
>
> 1. Lower overhead in spinning up a test, once the tests are already
> built (VIA a maven plugin)
> 2. I think the increase in complexity of JMeter comes with the benefit
> of extensibility (unless you know erlang, I guess..)
> 3. There is an existing community effort that can be leveraged to
> drive our performance tests [2]
>
> They both have exactly 3 advantages, I don't know what to do!?
>
> Just kidding. But, unless there is quantifiable evidence that suggests
> JMeter's performance will not suffice to properly test OAE (I have not
> been able to find such evidence yet, but others may have more data), I
> propose that we move forward with JMeter. I see value in making the
> JMeter tests executable from the same command-line on which OAE is
> built. I think this moves towards making the JMeter tests an artifact
> of the release and not some orthogonal set of scripts uploaded
> elsewhere, which in my experience tend to become of questionable age
> and relevance. I think it will become more valuable to our deployers,
> and the deployers' performance test data (which would hopefully be
> more abundant with the lower barrier of entry) will become more
> valuable to the core team.
>
> [1] https://github.com/kcampos/Open-Performance-Automation-Framework
> [2] https://confluence.sakaiproject.org/display/QA/CLE+Load+Test+Framework
>
> --
> Cheers,
> Branden
> _______________________________________________
> oae-dev mailing list
> [email protected]
> http://collab.sakaiproject.org/mailman/listinfo/oae-dev
>
--
Kyle Campos
Director of Quality Operations / rSmart
[email protected]
skype: kyle.campos
phone: 623-455-6180
GTalk: [email protected]
_______________________________________________
oae-dev mailing list
[email protected]
http://collab.sakaiproject.org/mailman/listinfo/oae-dev
