I like the command line action and simple editing of test scenarios. And the graphs look adequate in giving the client-side metrics and should be enough to show response time(s) and throughput. Of course the other side of this coin is monitoring server internals under load. I'm interested to see how Tsung does with the next couple steps in the process of load test creation, that is -
a. test script (virtual user) validation b. parameterization of the requests and other values I've found that the most critical aspect of authoring load tests is getting your virtual user error count to zero. And keeping it there is a trick. Otherwise, you have no idea whether errors which occur at run time are due to problems with the scripts or problems with the application. My biggest load testing headaches come during this parameterization and validation phase and also when old tests become stale. It's not the fault of the system under test, but something changed that cannot be foreseen that must be accounted for in the virtual user. The errors that occur due to problems with the virtual user itself don't look any different than typical web app errors. This adds test cycles and misdirected analysis (wild-goose chasing) for the tester, and even worse, for developers, if the results get back to them before you realize it's your tests that are the problem. Additionally, you can get a test script to validate cleanly when only 1 user is running, but as you scale out, more problems crop up. Finding the right values to parameterize is different for every app and often for each virtual user. The parameterization process is a bit like trying to find a needle in a haystack. There is a lot of trial and error. Even after applying this meticulous process, after a certain amount of time scripts just break and there is nothing you can do except re-record all over again. This is true for any load testing tool I suspect. I'm just saying it's critical to look at this aspect of the process and look into how each particular tool helps you get through it. That's just what I've seen in my experience. Other insights will be warmly welcomed. -Jon From: "Zach A. Thomas" <[email protected]<mailto:[email protected]>> Date: Tuesday, May 1, 2012 11:08 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [oae-dev] introduction to tsung for load tests Hi, everybody. Ever since rSmart shared the performance test automation work they've been doing[1] I've been itching to try it out. It's based on a tool call tsung[2], which is pretty similar to something like JMeter or the Grinder, but it seems simpler to use and it's written in erlang, so it excels at concurrency. It's my understanding that it can generate far more load from a small client machine than other tools. I made a couple of screencasts demonstrating it. The first one shows setting up the browser proxy, recording some behavior, and then starting a test: http://screencast.com/t/2jJOu3DDxtx (5 minutes) The second one is much shorter and just shows how to generate the reports (read pretty pictures) after a test run: http://screencast.com/t/mp9Ux2Y5vD (1 minute, 40) Kyle's code introduces an abstraction layer for generating the file that drives the tsung tests. Getting familiar with tsung is an important first step to doing that. Since we are becoming performance and scaling focused, I think it would be good for everybody on the team to be conversant with these tools. Comments or suggestions? regards, Zach [1] https://github.com/kcampos/Open-Performance-Automation-Framework [2] http://tsung.erlang-projects.org/
_______________________________________________ oae-dev mailing list [email protected] http://collab.sakaiproject.org/mailman/listinfo/oae-dev
