This sounds more like test strategy than a technology issue if I'm understanding your comments correctly. What I do in my tests is try to replicate a real user session as close as possible. In order to do that, you can't just do a record and playback. So while I use utilities to capture http traffic as a starting point there's a lot more work to do to make the tests re-usable, scalable, etc...
The random content widgets and AJAX polling requests require some additional logic to handle correctly. Which is the advantage of the ruby framework I wrote around the tool. It gives you a framework to do that, where a straight record/playback tool does not. Also there are some requests that don't belong in the test scripts, e.g. requests for assets that get cached on the client. -Kyle On Tue, May 1, 2012 at 1:51 PM, Cook, Jonathan <[email protected]> wrote: > by "problems with the virtual user" I meant an error which occurs post > recording. Meaning, there were no errors during the recording and now when > the test is re-run, there is an error. Problem with the VU as opposed to > problem with the app. For example, as I load tested OAE, I discovered that > as the landing page is loading there is a request to show some random > content: > > /var/search/public/random-content.json > > The result (in my recorded VU) is 10 items. Subsequently there are 10 > additional http requests for those items. All these requests are recorded > by the tool. For example /p/l4SC0Eiaa and 9 more like that. From what I > understand, those are very dynamic, so this recorded script has a lifespan. > It will start to fail if those items are deleted or if the dataset is > refreshed. So the errors in this case would be 404s. This is also what I > mean by a script that has the potential to become 'stale'. So I can either > delete those requests from the scripts or make the script read the results > of the random-content.json requests and dynamically construct the URLs. I > chose the latter, mainly because I want the virtual user to be as realistic > as possible. So in Tsung, it sounds like I would use dynamic variables to > accomplish this. > > An example of a script failing as you ramp up from 1 to many users is a > page similar to the above, but instead of varying (random content) each > time you hit the landing page, it varies by user. So the script will work > when it logs in as the (recorded) test user, and works during validation, > but fails during test execution as other test users login. You get 500 > errors in this case and stack traces about lack of permission in the server > logs. > > My point though is that I didn't know these things about OAE when I > began. And also that, although a tool can't solve these problems, it's > really awesome if it has clever ways of assisting you in discovering them. > The tool I use has some neat tricks for this, as an example mind you, not > as an option, as it is commercial and doesn't meet a key goal, that of > collaborating on test writing across the OAE community. Perhaps Tsung has a > different strategy that preempts this. I'm not aware that The Grinder or > Jmeter have this feature. But maybe the need for clever shortcuts isn't > really necessary if we're talking about using a load testing tool as > developers, using it in a way similar to unit testing. Perhaps I've become > a pampered load tester. As developers we will have this knowledge upfront > and will know what 'buttons' to push. I'm looking forward to test-driving > Tsung. Sorry this was so wordy. > > -Jon > > From: Kyle Campos <[email protected]> > Date: Tuesday, May 1, 2012 3:20 PM > To: Jonathan Cook <[email protected]> > Cc: "Zach A. Thomas" <[email protected]>, " > [email protected]" <[email protected]> > Subject: Re: [oae-dev] introduction to tsung for load tests > > Comments inline... > > On Tue, May 1, 2012 at 11:32 AM, Cook, Jonathan <[email protected]>wrote: > >> I like the command line action and simple editing of test scenarios. >> And the graphs look adequate in giving the client-side metrics and should >> be enough to show response time(s) and throughput. Of course the other side >> of this coin is monitoring server internals under load. I'm interested to >> see how Tsung does with the next couple steps in the process of load test >> creation, that is - >> >> a. test script (virtual user) validation >> b. parameterization of the requests and other values >> >> I've found that the most critical aspect of authoring load tests is >> getting your virtual user error count to zero. And keeping it there is a >> trick. Otherwise, you have no idea whether errors which occur at run time >> are due to problems with the scripts or problems with the application. >> > > I think this is more a question of test execution strategy. I run all > load use cases through with debug (dumptraffic=true) with a single session > before applying any load. I evaluate all the requests and responses to make > sure parameter substitution is working, dynamic variables are catching the > correct response data, etc... If the use case passes correctly with 1 > session, then any failures with multiple sessions should be due to load and > not functional problems with the script. This strategy has worked pretty > well. > > >> My biggest load testing headaches come during this parameterization and >> validation phase and also when old tests become stale. It's not the fault >> of the system under test, but something changed that cannot be foreseen >> that must be accounted for in the virtual user. The errors that occur due >> to problems with the virtual user itself don't look any different than >> typical web app errors. >> > > This sounds like a symptom of the platform you're using. I don't know > what you mean exactly with "problems with the virtual user". In tsung we're > just dealing with http requests that are collected in "transactions" that > are then collected in a session. The number of "users" is the number of > sessions you configure. > > >> This adds test cycles and misdirected analysis (wild-goose chasing) for >> the tester, and even worse, for developers, if the results get back to them >> before you realize it's your tests that are the problem. Additionally, you >> can get a test script to validate cleanly when only 1 user is running, but >> as you scale out, more problems crop up. Finding the right values to >> parameterize is different for every app and often for each virtual user. >> > > This is where dynamic variables come in handy. You can look at the > authentication.rb[1] module in my framework to see how I use them. This > grabs session(or user) specific data from http response data via regex and > stores it just for that session. > > The parameterization process is a bit like trying to find a needle in a >> haystack. There is a lot of trial and error. Even after applying this >> meticulous process, after a certain amount of time scripts just break and >> there is nothing you can do except re-record all over again. >> > > The only time I do any sort of "re-recording" process is to make sure > from major release to major release that I make sure all the existing > requests are still valid and I add any new requests that the existing test > cases may be missing(new features etc..). I don't blow away everything and > start over. I just update what's there on an as needed basis. > > >> This is true for any load testing tool I suspect. I'm just saying it's >> critical to look at this aspect of the process and look into how each >> particular tool helps you get through it. That's just what I've seen in my >> experience. Other insights will be warmly welcomed. >> > > Let me know if you have any questions about the implementation of > dynamic variables. It's proven to be pretty robust for me, occasionally I > have to adjust the regexp if the http response format has changed. I've > done that maybe once for OAE at this point. > > [1] > https://github.com/kcampos/Open-Performance-Automation-Framework/blob/master/lib/oae/common/authentication.rb(Line > 177-186) > > Thanks > -Kyle > > >> -Jon >> >> >> From: "Zach A. Thomas" <[email protected]> >> Date: Tuesday, May 1, 2012 11:08 AM >> To: "[email protected]" <[email protected]> >> >> Subject: [oae-dev] introduction to tsung for load tests >> >> Hi, everybody. Ever since rSmart shared the performance test automation >> work they've been doing[1] I've been itching to try it out. It's based on a >> tool call tsung[2], which is pretty similar to something like JMeter or the >> Grinder, but it seems simpler to use and it's written in erlang, so it >> excels at concurrency. It's my understanding that it can generate far more >> load from a small client machine than other tools. >> >> I made a couple of screencasts demonstrating it. The first one shows >> setting up the browser proxy, recording some behavior, and then starting a >> test: >> http://screencast.com/t/2jJOu3DDxtx (5 minutes) >> >> The second one is much shorter and just shows how to generate the >> reports (read pretty pictures) after a test run: >> http://screencast.com/t/mp9Ux2Y5vD (1 minute, 40) >> >> Kyle's code introduces an abstraction layer for generating the file >> that drives the tsung tests. Getting familiar with tsung is an important >> first step to doing that. Since we are becoming performance and scaling >> focused, I think it would be good for everybody on the team to be >> conversant with these tools. >> >> Comments or suggestions? >> >> regards, >> Zach >> >> [1] https://github.com/kcampos/Open-Performance-Automation-Framework >> [2] http://tsung.erlang-projects.org/ >> >> _______________________________________________ >> oae-dev mailing list >> [email protected] >> http://collab.sakaiproject.org/mailman/listinfo/oae-dev >> >> > > > -- > Kyle Campos > Director of Quality Operations / rSmart > [email protected] > skype: kyle.campos > phone: 623-455-6180 > GTalk: [email protected] > > -- Kyle Campos Director of Quality Operations / rSmart [email protected] skype: kyle.campos phone: 623-455-6180 GTalk: [email protected]
_______________________________________________ oae-dev mailing list [email protected] http://collab.sakaiproject.org/mailman/listinfo/oae-dev
