by "problems with the virtual user" I meant an error which occurs post recording. Meaning, there were no errors during the recording and now when the test is re-run, there is an error. Problem with the VU as opposed to problem with the app. For example, as I load tested OAE, I discovered that as the landing page is loading there is a request to show some random content:
/var/search/public/random-content.json The result (in my recorded VU) is 10 items. Subsequently there are 10 additional http requests for those items. All these requests are recorded by the tool. For example /p/l4SC0Eiaa and 9 more like that. From what I understand, those are very dynamic, so this recorded script has a lifespan. It will start to fail if those items are deleted or if the dataset is refreshed. So the errors in this case would be 404s. This is also what I mean by a script that has the potential to become 'stale'. So I can either delete those requests from the scripts or make the script read the results of the random-content.json requests and dynamically construct the URLs. I chose the latter, mainly because I want the virtual user to be as realistic as possible. So in Tsung, it sounds like I would use dynamic variables to accomplish this. An example of a script failing as you ramp up from 1 to many users is a page similar to the above, but instead of varying (random content) each time you hit the landing page, it varies by user. So the script will work when it logs in as the (recorded) test user, and works during validation, but fails during test execution as other test users login. You get 500 errors in this case and stack traces about lack of permission in the server logs. My point though is that I didn't know these things about OAE when I began. And also that, although a tool can't solve these problems, it's really awesome if it has clever ways of assisting you in discovering them. The tool I use has some neat tricks for this, as an example mind you, not as an option, as it is commercial and doesn't meet a key goal, that of collaborating on test writing across the OAE community. Perhaps Tsung has a different strategy that preempts this. I'm not aware that The Grinder or Jmeter have this feature. But maybe the need for clever shortcuts isn't really necessary if we're talking about using a load testing tool as developers, using it in a way similar to unit testing. Perhaps I've become a pampered load tester. As developers we will have this knowledge upfront and will know what 'buttons' to push. I'm looking forward to test-driving Tsung. Sorry this was so wordy. -Jon From: Kyle Campos <[email protected]<mailto:[email protected]>> Date: Tuesday, May 1, 2012 3:20 PM To: Jonathan Cook <[email protected]<mailto:[email protected]>> Cc: "Zach A. Thomas" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [oae-dev] introduction to tsung for load tests Comments inline... On Tue, May 1, 2012 at 11:32 AM, Cook, Jonathan <[email protected]<mailto:[email protected]>> wrote: I like the command line action and simple editing of test scenarios. And the graphs look adequate in giving the client-side metrics and should be enough to show response time(s) and throughput. Of course the other side of this coin is monitoring server internals under load. I'm interested to see how Tsung does with the next couple steps in the process of load test creation, that is - a. test script (virtual user) validation b. parameterization of the requests and other values I've found that the most critical aspect of authoring load tests is getting your virtual user error count to zero. And keeping it there is a trick. Otherwise, you have no idea whether errors which occur at run time are due to problems with the scripts or problems with the application. I think this is more a question of test execution strategy. I run all load use cases through with debug (dumptraffic=true) with a single session before applying any load. I evaluate all the requests and responses to make sure parameter substitution is working, dynamic variables are catching the correct response data, etc... If the use case passes correctly with 1 session, then any failures with multiple sessions should be due to load and not functional problems with the script. This strategy has worked pretty well. My biggest load testing headaches come during this parameterization and validation phase and also when old tests become stale. It's not the fault of the system under test, but something changed that cannot be foreseen that must be accounted for in the virtual user. The errors that occur due to problems with the virtual user itself don't look any different than typical web app errors. This sounds like a symptom of the platform you're using. I don't know what you mean exactly with "problems with the virtual user". In tsung we're just dealing with http requests that are collected in "transactions" that are then collected in a session. The number of "users" is the number of sessions you configure. This adds test cycles and misdirected analysis (wild-goose chasing) for the tester, and even worse, for developers, if the results get back to them before you realize it's your tests that are the problem. Additionally, you can get a test script to validate cleanly when only 1 user is running, but as you scale out, more problems crop up. Finding the right values to parameterize is different for every app and often for each virtual user. This is where dynamic variables come in handy. You can look at the authentication.rb[1] module in my framework to see how I use them. This grabs session(or user) specific data from http response data via regex and stores it just for that session. The parameterization process is a bit like trying to find a needle in a haystack. There is a lot of trial and error. Even after applying this meticulous process, after a certain amount of time scripts just break and there is nothing you can do except re-record all over again. The only time I do any sort of "re-recording" process is to make sure from major release to major release that I make sure all the existing requests are still valid and I add any new requests that the existing test cases may be missing(new features etc..). I don't blow away everything and start over. I just update what's there on an as needed basis. This is true for any load testing tool I suspect. I'm just saying it's critical to look at this aspect of the process and look into how each particular tool helps you get through it. That's just what I've seen in my experience. Other insights will be warmly welcomed. Let me know if you have any questions about the implementation of dynamic variables. It's proven to be pretty robust for me, occasionally I have to adjust the regexp if the http response format has changed. I've done that maybe once for OAE at this point. [1] https://github.com/kcampos/Open-Performance-Automation-Framework/blob/master/lib/oae/common/authentication.rb (Line 177-186) Thanks -Kyle -Jon From: "Zach A. Thomas" <[email protected]<mailto:[email protected]>> Date: Tuesday, May 1, 2012 11:08 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [oae-dev] introduction to tsung for load tests Hi, everybody. Ever since rSmart shared the performance test automation work they've been doing[1] I've been itching to try it out. It's based on a tool call tsung[2], which is pretty similar to something like JMeter or the Grinder, but it seems simpler to use and it's written in erlang, so it excels at concurrency. It's my understanding that it can generate far more load from a small client machine than other tools. I made a couple of screencasts demonstrating it. The first one shows setting up the browser proxy, recording some behavior, and then starting a test: http://screencast.com/t/2jJOu3DDxtx (5 minutes) The second one is much shorter and just shows how to generate the reports (read pretty pictures) after a test run: http://screencast.com/t/mp9Ux2Y5vD (1 minute, 40) Kyle's code introduces an abstraction layer for generating the file that drives the tsung tests. Getting familiar with tsung is an important first step to doing that. Since we are becoming performance and scaling focused, I think it would be good for everybody on the team to be conversant with these tools. Comments or suggestions? regards, Zach [1] https://github.com/kcampos/Open-Performance-Automation-Framework [2] http://tsung.erlang-projects.org/ _______________________________________________ oae-dev mailing list [email protected]<mailto:[email protected]> http://collab.sakaiproject.org/mailman/listinfo/oae-dev -- Kyle Campos Director of Quality Operations / rSmart [email protected]<mailto:[email protected]> skype: kyle.campos phone: 623-455-6180 GTalk: [email protected]<mailto:[email protected]>
_______________________________________________ oae-dev mailing list [email protected] http://collab.sakaiproject.org/mailman/listinfo/oae-dev
