[wtr-general] Re: Performance Testing
At the company where I work using oss tools for performance testing (www.altentee.com) we often use JMeter to 'spin up the wheels' or as Chuck put it, conduct 'protocol' level testing. At the same time we also run a quasi-smoke test if you like, using Watir to conduct 'browser' level testing. Protocol level tests do apply load differently albeit in subtle ways. Things like re-use or discontinued use of http connections, parallel execution of browser threads, client render time (DOM tree parsing and javascript processing) sometimes have an effect on end user response time. I have seen peculiar behavior of browsers (particularly IE) when the SUT is under load or when emulating network conditions (latency bandwidth). Things like '1 items remaining' or hung AJAX calls or even large discrepancies b/w protocol level response time and *actual* browser level response times... That's why we see significant value in using Watir + JMeter/the Grinder/OpenSTA at the same time. Operations such as BrowserMob (using selenium to control ur botnet) is useful in the sense it does sometimes simplify the script effort for your load test. We generally script 80% of the SUT with JMeter, but the other 20% often complicated by pesky AJAX controls and the like are sometimes easier done in Selenium and/or Watir. Another great use of Watir is to generate your load scripts in JMeter from Watir scripts. It sounds like overkill, but if you know you will be re-scripting again and again due to successive code drops etc, it's often easier to generate maintain these from Watir scripts i.e. playback watir with the JMeter/Badboy proxy/recorder after each code drop. Once you get familiar with the SUT you generally have a template bunch of regex extractors etc to apply to your freshly generated load scripts. I prefer to read Watir scripts rather than look at .jmx or .c files... Regards, Tim Koopmans On Apr 6, 9:21 am, Chuck van der Linden sqa...@gmail.com wrote: On Apr 3, 2:56 am, sai saidesertrose2...@gmail.com wrote: Even if you use http clients to generate requests and load test the app it is advisable to use real browsers to do some kind of load testing because thats how the app is going to be used in the end through real internet in the middle. Yes, that's how it will be 'used' but how can the server tell? The point of a loadtest is to see how your servers hold up under load, that load takes the form of http requests.. How exactly is the server supposed to be able to tell the difference between requests generated by a browerser, or a protocol evel tool that emulated the requests sent by the browser? I fail to see what value you are adding for the expense of having to maintain scripts for two different platforms (protocol level vs browser level). If you REALLY want to experience the 'user experience' while the test is running, just interact manually with a browser. The approach I take is when we are developing a functionality in an iteration I tend to use something like http clients to find bottlenecks and correct them. This gives me a fast feedback on parts I need to concentrate on. But once the app is in good shape where I can use it I tend to prefer using real browsers to test it along with http clients. The advantage of real browser approach is it is how the app will be used as well it takes away lots of complexities like ajax processing, streaming away from you. Again I fail to see the advantage.. emulating ajax processing doesn't matter, all that matters is what kind of post or get resuts when that processing is done and the user responds to it. If there's a delay due to the ajax or the user 'reading' or filling in a form or whatever, that's what Think Time is for. Steaming can be trickier I'll admit, but there are some things out there for testing flash and such that will let you start up a large number of streams and maintain them for however long you want. But ultimately we get back to the question of 'how can the server tell the difference' because if it can, then you have an issue with your loadtest tool, and if it can't why bother with the browser? The disadvantage is it is not possible to run more than three browsers at a time as they are resource heavy. There's also the expense of maintaining two sets of scripts/tests. And so clouds may help to scale in this issue. IF you have a suitable connection to the intenet that you can afford to load up with cloud traffic perhaps.. If you are running a production site you dare not clog it's pipe with loadtest traffic. So that means you need two more pipes one for the cloud load to arrive on, and one for a 'back channel' that you use to control the cloud, and receive instrumentation regarding response times, lost packets or failed responses, and perhaps telemetry as to what the scripts are doing at a particuar moment.. which you wil then need to correlate with the instrumentation of the
[wtr-general] Re: Performance Testing
I think that Paul already mentioned this, but I started down the road of creating a watir script that emulated what a user does on our web site, then running that script on hundreds of virtual machines using Amazon's EC2 service. The only reason that I didn't actually carry out the test is because the testing responsibility changed to someone other than me. If you have your watir script(s) created, this is pretty easy and inexpensive--at least to drive the load. Andrew McFarlane Date: Thu, 2 Apr 2009 14:37:21 -0600 Subject: [wtr-general] Re: Performance Testing From: lisa.cris...@gmail.com To: watir-general@googlegroups.com We have done a baseline with JMeter, we like the tool, but we used BadBoy to create the scripts (and it was a fairly rudimentary test). I was just wondering the other day if it's possible to incorporate Watir scripts with some perf/load test tool. I heard BrowserMob can use Selenium scripts. -- Lisa On Thu, Apr 2, 2009 at 2:21 PM, Jason Shelton jas.shel...@hotmail.com wrote: All, I am writing to ask the group what tools you all are using for performance/load testing on your web apps. Are you using Watir? If so, how did you structure your tests to simulate multiple users? Thanks in advance for all responses. - Shelton Quick access to your favorite MSN content and Windows Live with Internet Explorer 8. Download FREE now! -- Lisa Crispin Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers and Agile Teams_ (Addison-Wesley 2009) http://lisacrispin.com _ Windows Live™: Keep your life in sync. http://windowslive.com/explore?ocid=TXT_TAGLM_WL_allup_1a_explore_042009 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Watir General group. To post to this group, send email to watir-general@googlegroups.com Before posting, please read the following guidelines: http://wiki.openqa.org/display/WTR/Support To unsubscribe from this group, send email to watir-general-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/watir-general -~--~~~~--~~--~--~---
[wtr-general] Re: Performance Testing
Andrew, how easy was it to set up the ami with a watir environment? How did you control running the scripts? Paul On Fri, Apr 3, 2009 at 1:22 PM, Andrew McFarlane welkin_...@hotmail.comwrote: I think that Paul already mentioned this, but I started down the road of creating a watir script that emulated what a user does on our web site, then running that script on hundreds of virtual machines using Amazon's EC2 service. The only reason that I didn't actually carry out the test is because the testing responsibility changed to someone other than me. If you have your watir script(s) created, this is pretty easy and inexpensive--at least to drive the load. Andrew McFarlane -- Date: Thu, 2 Apr 2009 14:37:21 -0600 Subject: [wtr-general] Re: Performance Testing From: lisa.cris...@gmail.com To: watir-general@googlegroups.com We have done a baseline with JMeter, we like the tool, but we used BadBoy to create the scripts (and it was a fairly rudimentary test). I was just wondering the other day if it's possible to incorporate Watir scripts with some perf/load test tool. I heard BrowserMob can use Selenium scripts. -- Lisa On Thu, Apr 2, 2009 at 2:21 PM, Jason Shelton jas.shel...@hotmail.comwrote: All, I am writing to ask the group what tools you all are using for performance/load testing on your web apps. Are you using Watir? If so, how did you structure your tests to simulate multiple users? Thanks in advance for all responses. - Shelton -- Quick access to your favorite MSN content and Windows Live with Internet Explorer 8. Download FREE now! -- Lisa Crispin Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers and Agile Teams_ (Addison-Wesley 2009) http://lisacrispin.com -- Windows Live™: Keep your life in sync. Check it out.http://windowslive.com/explore?ocid=TXT_TAGLM_WL_allup_1a_explore_042009 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Watir General group. To post to this group, send email to watir-general@googlegroups.com Before posting, please read the following guidelines: http://wiki.openqa.org/display/WTR/Support To unsubscribe from this group, send email to watir-general-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/watir-general -~--~~~~--~~--~--~---
[wtr-general] Re: Performance Testing
We have done a baseline with JMeter, we like the tool, but we used BadBoy to create the scripts (and it was a fairly rudimentary test). I was just wondering the other day if it's possible to incorporate Watir scripts with some perf/load test tool. I heard BrowserMob can use Selenium scripts. -- Lisa On Thu, Apr 2, 2009 at 2:21 PM, Jason Shelton jas.shel...@hotmail.comwrote: All, I am writing to ask the group what tools you all are using for performance/load testing on your web apps. Are you using Watir? If so, how did you structure your tests to simulate multiple users? Thanks in advance for all responses. - Shelton -- Quick access to your favorite MSN content and Windows Live with Internet Explorer 8. Download FREE now!http://ie8.msn.com/microsoft/internet-explorer-8/en-us/ie8.aspx?ocid=B037MSN55C0701A -- Lisa Crispin Co-author with Janet Gregory, _Agile Testing: A Practical Guide for Testers and Agile Teams_ (Addison-Wesley 2009) http://lisacrispin.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Watir General group. To post to this group, send email to watir-general@googlegroups.com Before posting, please read the following guidelines: http://wiki.openqa.org/display/WTR/Support To unsubscribe from this group, send email to watir-general-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/watir-general -~--~~~~--~~--~--~---
[wtr-general] Re: Performance Testing
On Apr 2, 2:01 pm, Paul Rogers paul.rog...@shaw.ca wrote: most load tools do direct http posts/gets/put requests. jmeter does this using a java library, to get the high levels of virtual users. Browsermob uses amazon cloud running ms windows 2003 server and a real browser. Paul is absolutely right in this regard. This is because the overhead of rendering and running client side code means if you use anything remotely resembling a 'real browser' it won't scale well, and you will need a small fleet of systems to generate the load. It's also un-necessary. some vendors who don't support protocol level load will try to justify their tool as being 'more realistic' which I say is utter one-hundred percent hogwash. The server has NO idea who is sending those packets, and it can't tell IE or Firefox from a good tool claiming to be IE or Firefox.. the 'realism' is all in the mind of the salesman because the web server hasn't a clue. Ive think the best way of doing load tests is to use jmeter or other tool. You may be able to record a script via the jmeter proxy when watir is running. I can see a lot of advantages to this, but it takes a lot of hand editing to go and make the recorded jmeter script usable for more than one user. Ive found load testing to be a particulary difficult thing todo. Mostly because of the question what are we trying to do here? - find out what bit breaks first? Find out how many users we can support before we need a new bit of hardware? Find out what the slowest operastion is? Find out how to make the slowest operation better? There is a book that Scott Barber was a coauthor of, that is available as a free pdf that is good reading. I agree 100% Paul. load testing and perf testing is some of the more 'scientific' testing you can do, it requires close attention to detail, tight control over variables, tests that are highly reproducable, and as you point out you have to start out by asking the right question. A loadtest is basically an experiment, and like any experiment it's only going to be as successful as the hypothesis it's trying to prove or disprove. If you don't know the question you are trying to ask, you're not likely to design loadtest that will give you results you can use. As for myself, I've used both loadrunner (really good but really expensive) and Microsoft's Visual Studio Team System2008 (both the test edition and team suite versions have a HTTP level web-test and loadtest capability) I found in my work that it scales well and blows loadrunner away in terms of bang for the buck. I've used it with just a single machine as the load generator (also running the controller as wel) and brought both webservers and db servers to their virtual knees and had they crying uncle! There's also an excellent MSDN forum with some of the main pm's etc providing advice and support, and a small collection of videos and such. The downside is that it does all the SUT monitoring via WMI (Microsoft's implementation of WBEM) which works fabulous on windows servers, but because the *nix community never implemented WBEM (a multi vendor DMTF standard) there's no performance monitoring daemon that allows you to connect to the machine using WBEM, and hence you can't monitor the performance of non windows servers, and without good (really good) data on how the systems are being affected by the load, corrilated with what the scripts are doing, the loadtests are of much less value. Sorry but this is a bit of a soapbox for me. it's really ironic in a way that the closed' windows OS, because MS implemented an open standard is more accessable in terms of remote monitoring and rathering of system information (out of the box without having to installl anything, and merely configuring some accounts that are allowed to access the data) than the supposedly 'open' os's which never adopted that standdard and are thus a relative black hole in terms of being able to remotely monitor performance counters --Chuck --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Watir General group. To post to this group, send email to watir-general@googlegroups.com Before posting, please read the following guidelines: http://wiki.openqa.org/display/WTR/Support To unsubscribe from this group, send email to watir-general-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/watir-general -~--~~~~--~~--~--~---
