Comments inline... On Tue, May 1, 2012 at 11:32 AM, Cook, Jonathan <[email protected]>wrote:
> I like the command line action and simple editing of test scenarios. And > the graphs look adequate in giving the client-side metrics and should be > enough to show response time(s) and throughput. Of course the other side of > this coin is monitoring server internals under load. I'm interested to see > how Tsung does with the next couple steps in the process of load test > creation, that is - > > a. test script (virtual user) validation > b. parameterization of the requests and other values > > I've found that the most critical aspect of authoring load tests is > getting your virtual user error count to zero. And keeping it there is a > trick. Otherwise, you have no idea whether errors which occur at run time > are due to problems with the scripts or problems with the application. > I think this is more a question of test execution strategy. I run all load use cases through with debug (dumptraffic=true) with a single session before applying any load. I evaluate all the requests and responses to make sure parameter substitution is working, dynamic variables are catching the correct response data, etc... If the use case passes correctly with 1 session, then any failures with multiple sessions should be due to load and not functional problems with the script. This strategy has worked pretty well. > My biggest load testing headaches come during this parameterization and > validation phase and also when old tests become stale. It's not the fault > of the system under test, but something changed that cannot be foreseen > that must be accounted for in the virtual user. The errors that occur due > to problems with the virtual user itself don't look any different than > typical web app errors. > This sounds like a symptom of the platform you're using. I don't know what you mean exactly with "problems with the virtual user". In tsung we're just dealing with http requests that are collected in "transactions" that are then collected in a session. The number of "users" is the number of sessions you configure. > This adds test cycles and misdirected analysis (wild-goose chasing) for > the tester, and even worse, for developers, if the results get back to them > before you realize it's your tests that are the problem. Additionally, you > can get a test script to validate cleanly when only 1 user is running, but > as you scale out, more problems crop up. Finding the right values to > parameterize is different for every app and often for each virtual user. > This is where dynamic variables come in handy. You can look at the authentication.rb[1] module in my framework to see how I use them. This grabs session(or user) specific data from http response data via regex and stores it just for that session. The parameterization process is a bit like trying to find a needle in a > haystack. There is a lot of trial and error. Even after applying this > meticulous process, after a certain amount of time scripts just break and > there is nothing you can do except re-record all over again. > The only time I do any sort of "re-recording" process is to make sure from major release to major release that I make sure all the existing requests are still valid and I add any new requests that the existing test cases may be missing(new features etc..). I don't blow away everything and start over. I just update what's there on an as needed basis. > This is true for any load testing tool I suspect. I'm just saying it's > critical to look at this aspect of the process and look into how each > particular tool helps you get through it. That's just what I've seen in my > experience. Other insights will be warmly welcomed. > Let me know if you have any questions about the implementation of dynamic variables. It's proven to be pretty robust for me, occasionally I have to adjust the regexp if the http response format has changed. I've done that maybe once for OAE at this point. [1] https://github.com/kcampos/Open-Performance-Automation-Framework/blob/master/lib/oae/common/authentication.rb(Line 177-186) Thanks -Kyle > -Jon > > > From: "Zach A. Thomas" <[email protected]> > Date: Tuesday, May 1, 2012 11:08 AM > To: "[email protected]" <[email protected]> > > Subject: [oae-dev] introduction to tsung for load tests > > Hi, everybody. Ever since rSmart shared the performance test automation > work they've been doing[1] I've been itching to try it out. It's based on a > tool call tsung[2], which is pretty similar to something like JMeter or the > Grinder, but it seems simpler to use and it's written in erlang, so it > excels at concurrency. It's my understanding that it can generate far more > load from a small client machine than other tools. > > I made a couple of screencasts demonstrating it. The first one shows > setting up the browser proxy, recording some behavior, and then starting a > test: > http://screencast.com/t/2jJOu3DDxtx (5 minutes) > > The second one is much shorter and just shows how to generate the > reports (read pretty pictures) after a test run: > http://screencast.com/t/mp9Ux2Y5vD (1 minute, 40) > > Kyle's code introduces an abstraction layer for generating the file that > drives the tsung tests. Getting familiar with tsung is an important first > step to doing that. Since we are becoming performance and scaling focused, > I think it would be good for everybody on the team to be conversant with > these tools. > > Comments or suggestions? > > regards, > Zach > > [1] https://github.com/kcampos/Open-Performance-Automation-Framework > [2] http://tsung.erlang-projects.org/ > > _______________________________________________ > oae-dev mailing list > [email protected] > http://collab.sakaiproject.org/mailman/listinfo/oae-dev > > -- Kyle Campos Director of Quality Operations / rSmart [email protected] skype: kyle.campos phone: 623-455-6180 GTalk: [email protected]
_______________________________________________ oae-dev mailing list [email protected] http://collab.sakaiproject.org/mailman/listinfo/oae-dev
