Requirements

[Jordi Salvat i Alabart]

This is a message I wrote a while ago to justify my decision to use JMeter as our load-testing tool for my job. All references to customers and colleagues removed, of course. Notes in [] have been added now.

I'll now work on extracting some high-level requirements for JMeter, but I thought I would share this because of the comparison with other tools it contains. (USAns would call this "competitive intelligence", I think).

�

REQUIRED:

1.- Portability and ease of installation: should be able to get up and
running on any customer site in a few hours. Since Java is available
wherever our app is running, 100% Java solutions are strongly preferred.
[should have added it should work with any of the usual JDKs: 1.2.2 to 1.4.1 -- and in all O.S.s we support: win32, linux, Solaris, AIX, HP-UX]

2.- Simplicity and ease of use: I need to be able to create test scripts
and run them in minutes, not days. Navigation needs to be based on URLs,
not on link-text or link-position-in-page. Sniffer or proxy for script
creation a big plus.

3.- Support for all common stuff (GET, POST,...), including HTTPS.

4.- Possibility to have each thread use different data (tipically
username & password) on form submissions in a general-enough way.

5.- Shouldn't break on illegal redirects (as [our current tool] does) and other illegal but common stuff.

6.- Either a command-line interface or a distributed load generation
mechanism which is easy to set up even in the presence of firewalls
(through ssh tunneling and alike).

WISHED:

A.- Good performance: I want the threshold for needing to set up
distributed load generation to be as high as possible.

B.- A couple nice graphs to show to the customer.

With these requirements in mind, and having mostly discarded the
commercial big-players (mainly because of req. 2), I had a look at the
following opensourced tools:

ab [part of the Apache distribution]
guts
siege
openload
Push-to-test Load / TestMaker
Jakarta JMeter
Grinder
deluge

Actually, I discarded all of them as not fulfilling my requirements. The
finalists (with my requirements in mind: ab and siege are very good
tools, just not what I need) were:

- Grinder
Its main problem is that it requires multicast networking. I'm scared
this may be an unsurmountable difficulty on the random customer site.

- Deluge
Performance is poor. Portability also limited: uses Perl and requires a
good deal of Perl packages which is not commonly installed in customer's
sites.

- Push-to-test TestMaker (Load is old and will probably be deprecated soon)
Just too difficult to create scripts. Need to learn Python, and I'm too
lazy. (Don't be confused: it's 100% Java, only it uses an embeded Python
interpreter written in Java for scripting). I couldn't find a
proxy/sniffer for easy script creation.

- Jakarta-JMeter
Still evaluating this one, since it looks the most promising. Has some
problems with the handling of redirects on the script-creation
proxy/sniffer, but I'm in touch with the developers (the Apache group)
to solve this and continue my evaluation. The fact that it's part of the
Apache group gives it good opportunity to gather momentum, so it's
probably a safe bet in the mid term. [How mistaken I was at that point! -- looks like it's getting there, though.]

BTW, there's dozens -- probably hundreds -- of similar tools out there.
Looks like every web app developper has at some point built his own,
just as we did with [our current tool] :-)

Salut,

Jordi.

�



--
To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>