What exactly are you trying to measure? User experience, server capability, browser performance, etc?
There are obviously many aspects of load testing, and each serves a different purpose. It seems you have started in one direction, and changed path along the way. What statistics do you want to produce, in order to show what about your system? Charlie > -----Original Message----- > From: Zippy Zeppoli [mailto:[email protected]] > Sent: Thursday, 7 February 2013 10:33 AM > To: JMeter Users List > Subject: Re: complex javascript actions in jmeter load test > > There seems to be a big misunderstanding of what is the benefit of using a > browser. > I think this thread at this point is far off topic, although it looks like a > lot of > good knowledge was shaken out of it, so I will consider it time well spent, > and a question well asked. > > And now the clarification: > > Comparing RUM & Synthetic Page Load Times > > November 14, 2012 5:30 pm | 13 Comments > > Yesterday I read Etsy's October 2012 Site Performance Report. Etsy is one of > only a handful of companies that publish their performance stats with > explanations and future plans. It's really valuable (and brave!), and gives > other developers an opportunity to learn from an industry leader. In this > article Etsy mentions that the page load time stats are gathered from a > private instance of WebPagetest. They explain their use of synthetically- > generated measurements instead of RUM (Real User Monitoring) data: > > You might be surprised that we are using synthetic tests for this front-end > report instead of Real User Monitoring (RUM) data. RUM is a big part of > performance monitoring at Etsy, but when we are looking at trends in front- > end performance over time, synthetic testing allows us to eliminate much of > the network variability that is inherent in real user data. This helps us tie > performance regressions to specific code changes, and get a more stable > view of performance overall. > > Etsy's choice of synthetic data for tracking performance as part of their > automated build process totally makes sense. I've talked to many companies > that do the same thing. Teams dealing with builds and code regressions > should definitely do this. BUT... it's important to include RUM data when > sharing performance measurements beyond the internal devops team. > > Why should RUM data always be used when talking beyond the core team? > > The issue with only showing synthetic data is that it typically makes a > website > appear much faster than it actually is. This has been true since I first > started > tracking real user metrics back in 2004. My rule-of-thumb is that your real > users are experiencing page load times that are twice as long as their > corresponding synthetic measurements. > > From http://www.stevesouders.com/blog/2012/11/14/comparing-rum- > synthetic-page-load-times/ > > > > On Wed, Feb 6, 2013 at 4:30 PM, Zippy Zeppoli <[email protected]> > wrote: > > > > I think you may be missing the point. > > Real load cannot be tested via HTTP interactions. > > There is no DOM rendering happening. > > I can make HTTP requests all day and it won't reflect the true response time > unless it's done through a browser. > > > > Recording a script in Jmeter proxy is trivial. Simulating *real* user load > > is > not it requires a browser and interactions with a web application. > > > > > > On Tue, Feb 5, 2013 at 6:51 PM, Deepak Shetty <[email protected]> > wrote: > >> > >> >Actually that does matter it cannot do JavaScript. If a request > >> >requires you need to be able to click a JavaScript button then the > >> >request will never happen. > >> The point is that what happens when the button is clicked? Assuming > >> its a server - ajax call then A HTTP call is made and some parameters > >> are passed and some values are returned. Thats whats important for > >> the load test , not the fact that javascript was executed. > >> So when you record the script , you will be the person clicking the > >> button(you are recording your actions) , JMeter will record every > >> interaction that makes a call to the server and will record this as a > >> separate HTTP request and when you run the script the same request > >> will be made as if someone clicked the button! > >> > >> You dont need to use the recorder either , you can modify the script > >> yourself. > >> > >> If the javascript didnt actually make any server side call - then it > >> doesnt matter because you dont want to load test this anyway. > >> > >> Have you actually tried this? It sounds as if you have a problem > >> recording your script and you probably have concluded that JMeter > >> doesnt do javascript (true) and hence cant test websites that do > >> javascript/ajax > >> (false) > >> > >> >Real browser is needed > >> Not for a good deal of use cases - as many of the people on this > >> mailing list can attest too. > >> > >> > >> > >> > >> > >> On Tue, Feb 5, 2013 at 6:44 PM, Zippy Zeppoli > <[email protected]>wrote: > >> > >> > Deepak, > >> > Actually that does matter it cannot do JavaScript. If a request > >> > requires you need to be able to click a JavaScript button then the > >> > request will never happen. No request will ever be made. Also > >> > testing true web performance requires rendering the DOM, not just > >> > initiating HTTP requests and recording the response time, rps, etc. > >> > > >> > Real browser is needed, with JavaScript, and Jmeter doesn't > >> > integrate well with this, it isn't designed for this, which is > >> > understandable. The problem is there is a gap between real browser > >> > testing (owned by third party > >> > companies) and open source tools (Jmeter). There's nothing in > >> > between for real-browser based performance testing. I could go into > >> > why, but its off topic of this list, and I'd rather spare everyone the > >> > gas. > >> > > >> > Point being, Jmeter cannot solve my problem, without some serious > >> > customization. > >> > > >> > > >> > > >> > On Tue, Feb 5, 2013 at 6:33 PM, Deepak Shetty <[email protected]> > wrote: > >> > > >> > > Hi > >> > > You are getting too caught up in the JMeter doesnt do javascript thing. > >> > In > >> > > most cases it doesnt matter. > >> > > You have a webserver that is receiving HTTP requests - whether > >> > > those requests are generated via the user clicking a link or via > >> > > AJAX or via flash is hardly relevant to the webserver. It sees > >> > > HTTP requests and > >> > sends > >> > > HTTP responses. > >> > > JMeter deals with HTTP request and responses. As long as you can > >> > > make the same request that your javascript is making (which you > >> > > can see via the recording feature) , you can test it with Jmeter. > >> > > The fact that the proxy cant record javascript is irrelevant - it > >> > > only needs to record the requests. If your javascript is > >> > > responsible for generating the requests > >> > in > >> > > some way , then you have to replicate that within JMeter. > >> > > > >> > > Some caveats here are > >> > > a. Your toolkit should be good when you need to > >> > > parameterise(JQuery is great , GWT sucks) b. If you need to > >> > > perform some AJAX requests in parallel then JMeter isnt quite > >> > > there yet. > >> > > > >> > > Its probably true that it is easier writing scripts for Selenium > >> > > like > >> > tools > >> > > - but they also need more resources as well as given how often a > >> > > UI > >> > changes > >> > > as opposed to the HTTP request/response pair , sometimes JMeter > >> > > scripts > >> > are > >> > > more resilient to change. > >> > > > >> > > regards > >> > > deepak > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > On Tue, Feb 5, 2013 at 5:29 PM, Zippy Zeppoli > >> > > <[email protected] > >> > > >wrote: > >> > > > >> > > > Proxy won't work for clicking on JavaScript. > >> > > > Selenium will, however, Selenium isn't designed for performance > >> > testing, > >> > > > with the exception of BrowserMob, who has perfected it. > >> > > > > >> > > > If JMeter had some kind of (good) Selenium integration then it > >> > > > might be able to achieve it. > >> > > > > >> > > > I think JMeter is a good tool, but it's not built for this kind > >> > > > of > >> > work. > >> > > > > >> > > > On Tue, Feb 5, 2013 at 5:20 PM, David Luu <[email protected]> > wrote: > >> > > > > >> > > > > I'm not familiar with the full feature set of BrowserMob, but > >> > > > > JMeter > >> > > can > >> > > > do > >> > > > > what you want, easy or hard, depending on your needs. > >> > > > > > >> > > > > If all you need is simple record & playback, JMeter proxy is > >> > > > > best > >> > > option. > >> > > > > Just record w/ proxy, it generates a basic test plan that you > >> > > > > can > >> > save > >> > > to > >> > > > > file and run w/ X threads to generate load, etc. > >> > > > > > >> > > > > But that approach will always use the same user, same data in > >> > > > > the > >> > AJAX > >> > > > > calls during load generation. > >> > > > > > >> > > > > If you need them to be unique & parameterized, that's where > >> > > > > it takes > >> > > some > >> > > > > work to go in and cleanup the proxy recording w/ > >> > > > > paramaterized data, response checking (if more than just > >> > > > > checking HTTP 200 OKs). And > >> > you'll > >> > > > > find proxy recording generates a lot of unnecessary HTTP > >> > > > > requests for > >> > > > HTML, > >> > > > > CSS, image files that aren't necessary of interest as you're > >> > > > > testing > >> > > the > >> > > > > AJAX calls, which is where you filter out only the requests > >> > > > > of > >> > interest > >> > > > and > >> > > > > only use those in test plan, etc. > >> > > > > > >> > > > > I believe this kind of work is needed regardless of what tool > >> > > > > you > >> > use, > >> > > > some > >> > > > > tools just make it easier, but there's still work to do. Not > >> > > > > just a > >> > one > >> > > > > touch button approach unless all you need is simple record & > >> > playback. > >> > > > > > >> > > > > On Tue, Feb 5, 2013 at 5:08 PM, Zippy Zeppoli < > >> > [email protected] > >> > > > > >wrote: > >> > > > > > >> > > > > > Will probably just buy BrowserMob. > >> > > > > > Too bad there isn't an open source framework to already do this. > >> > > > > > > >> > > > > > Building this is a yak shave, and I need to be testing, not > >> > building > >> > > a > >> > > > > test > >> > > > > > harness thats probably going to break on the next release. > >> > > > > > > >> > > > > > On Tue, Feb 5, 2013 at 3:40 PM, David Luu > >> > > > > > <[email protected]> > >> > > wrote: > >> > > > > > > >> > > > > > > JMeter proxy is the most integrated approach, but I find > >> > > > > > > it gives > >> > > > "too > >> > > > > > much > >> > > > > > > information", so I tend to use external tools like > >> > > > > > > browser > >> > traffic > >> > > > > > sniffers > >> > > > > > > (HttpFox, livehttpheaders, ieHttpHeaders) to see what > >> > > > > > > HTTP > >> > requests > >> > > > are > >> > > > > > > made for AJAX calls for just the requests & responses I'm > >> > > interested > >> > > > > in, > >> > > > > > > easier to manage. But that's just my opinion, others > >> > > > > > > might find > >> > the > >> > > > > > JMeter > >> > > > > > > proxy easier to use. > >> > > > > > > > >> > > > > > > Whatever approach you take, you just need to know what > >> > > > > > > HTTP > >> > > requests > >> > > > > are > >> > > > > > > made by the AJAX calls to replicate in JMeter, and > >> > > > > > > parameterize > >> > > those > >> > > > > > > requests to take in dynamic/test data as needed, assert > >> > appropriate > >> > > > > > > response data, etc. In this case, WebDriver can be > >> > > > > > > dropped from > >> > the > >> > > > > > > equation, just gives more overhead and lowers scalability > >> > > > > > > in load generation. > >> > > > > > > > >> > > > > > > On Tue, Feb 5, 2013 at 1:55 PM, Philippe Mouawad < > >> > > > > > > [email protected]> wrote: > >> > > > > > > > >> > > > > > > > AjaxCall will emit a network call which JMeter will capture. > >> > > > > > > > > >> > > > > > > > Use JMeter Server Proxy and you should have the Ajax > >> > > > > > > > Calls > >> > > > recorded. > >> > > > > > > > > >> > > > > > > > Regards > >> > > > > > > > Philippe M. > >> > > > > > > > > >> > > > > > > > Follow me on twitter <https://twitter.com/philmdot> > >> > > > > > > > > >> > > > > > > > UBIK-INGENIERIE on TWITTER > >> > > > > > > > <https://twitter.com/ubikingenierie > >> > > > >> > > > > > > > > >> > > > > > > > UBIK LOAD PACK BLOG > >> > > > > > > > <http://www.ubik-ingenierie.com/blog/> > >> > > > > > > > > >> > > > > > > > On Tue, Feb 5, 2013 at 10:52 PM, Zippy Zeppoli < > >> > > > > [email protected] > >> > > > > > > > >wrote: > >> > > > > > > > > >> > > > > > > > > To clarify: > >> > > > > > > > > An example would be: > >> > > > > > > > > 1) log in via a form post > >> > > > > > > > > 2) look at orders in an ecommerce interface (AJAX > >> > > > > > > > > call) > >> > > > > > > > > 3) click on result to view order detail (AJAX) > >> > > > > > > > > > >> > > > > > > > > Pretty much a standard ecommerce transaction, but the > >> > interface > >> > > > is > >> > > > > > all > >> > > > > > > > > javascript. > >> > > > > > > > > > >> > > > > > > > > On Mon, Feb 4, 2013 at 7:40 PM, Stott, Charlie < > >> > > [email protected]> > >> > > > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > > > -----Original Message----- > >> > > > > > > > > > > From: David Luu [mailto:[email protected]] > >> > > > > > > > > > > Sent: Tuesday, 5 February 2013 1:18 PM > >> > > > > > > > > > > To: JMeter Users List > >> > > > > > > > > > > Subject: Re: complex javascript actions in jmeter > >> > > > > > > > > > > load > >> > test > >> > > > > > > > > > > > >> > > > > > > > > > > >> You can use webdriver from jmeter. Create a > >> > webdriver > >> > > > > class > >> > > > > > > that > >> > > > > > > > > > > performs the requests and runs the javascript via > >> > > > > > > > > > > the > >> > > > browser, > >> > > > > > then > >> > > > > > > > > > run/call > >> > > > > > > > > > > it from a BSF or JSR sampler. > >> > > > > > > > > > > > >> > > > > > > > > > > Just to clarify, I take it that's only worthwhile > >> > > > > > > > > > > to do > >> > (in > >> > > > > terms > >> > > > > > > of > >> > > > > > > > > > > scalability) when using PhantomJSDriver or > >> > > > > > > > > > > HtmlUnitDriver > >> > > or > >> > > > > > > > > > FirefoxDriver > >> > > > > > > > > > > (on Linux with xvfb) with JMeter this way? > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > "Worthwhile" depends on assumptions. > >> > > > > > > > > > > >> > > > > > > > > > > Because otherwise, the browser GUI is the > >> > > > > > > > > > > scalability > >> > > > limiting > >> > > > > > > factor > >> > > > > > > > > > even > >> > > > > > > > > > > with JMeter and Grid deployment, and in that > >> > > > > > > > > > > case, no > >> > > > > difference > >> > > > > > in > >> > > > > > > > > using > >> > > > > > > > > > > WebDriver outside JMeter to do performance tests > >> > > > > > > > > > > except > >> > if > >> > > > one > >> > > > > > > wants > >> > > > > > > > > the > >> > > > > > > > > > > JMeter logging/reporting facilities to help > >> > > > > > > > > > > performance > >> > > test, > >> > > > > > > because > >> > > > > > > > > > there's > >> > > > > > > > > > > no or minimal scalabiity gain. > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > Seems the poster is more at the > >> > > > > > > > > > capability/viability stage > >> > of > >> > > > > > > > developing > >> > > > > > > > > > tests? We would need much more information to > >> > > > > > > > > > start > >> > advising > >> > > > on > >> > > > > > the > >> > > > > > > > > entire > >> > > > > > > > > > load testing process, start to finish, and what > >> > > > > > > > > > challenges > >> > > may > >> > > > > face > >> > > > > > > the > >> > > > > > > > > > poster along the way. > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > On Mon, Feb 4, 2013 at 4:42 PM, Stott, Charlie < > >> > > > > [email protected]> > >> > > > > > > > > wrote: > >> > > > > > > > > > > > >> > > > > > > > > > > > You can use webdriver from jmeter. Create a > >> > > > > > > > > > > > webdriver > >> > > > class > >> > > > > > that > >> > > > > > > > > > > > performs the requests and runs the javascript > >> > > > > > > > > > > > via the > >> > > > > browser, > >> > > > > > > then > >> > > > > > > > > > > > run/call it from a BSF or JSR sampler. > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > -----Original Message----- > >> > > > > > > > > > > > > From: Zippy Zeppoli > >> > > > > > > > > > > > > [mailto:[email protected]] > >> > > > > > > > > > > > > Sent: Tuesday, 5 February 2013 9:28 AM > >> > > > > > > > > > > > > To: JMeter Users List > >> > > > > > > > > > > > > Subject: Re: complex javascript actions in > >> > > > > > > > > > > > > jmeter > >> > load > >> > > > test > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > The problem is Selenium has no performance > >> > > > > > > > > > > > > testing > >> > > > harness. > >> > > > > > > > > > > > > Sucks that it seems BrowserMob (paid > >> > > > > > > > > > > > > solution) is the > >> > > > only > >> > > > > > > solid > >> > > > > > > > > > option. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Until someone builds something with > >> > > > > > > > > > > > > Phantom.js, but > >> > it > >> > > > > seems > >> > > > > > > > JMeter > >> > > > > > > > > > > > > isn't going to cut it here. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > On Fri, Feb 1, 2013 at 5:40 PM, David Luu < > >> > > > > > [email protected]> > >> > > > > > > > > > wrote: > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > You'll need to figure out what the complex > >> > javascript > >> > > > > does. > >> > > > > > > > Does > >> > > > > > > > > > > > > > it make any AJAX requests, or is it all > >> > > > > > > > > > > > > > local > >> > client > >> > > > side > >> > > > > > > > > > > > processing/rendering? > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > If it's all local, then there's no point > >> > > > > > > > > > > > > > testing it > >> > > > with > >> > > > > > > > JMeter, > >> > > > > > > > > > > > > > that's client side browser testing better > >> > > > > > > > > > > > > > done with > >> > > > > > Selenium. > >> > > > > > > > It > >> > > > > > > > > > > > > > won't impact the server side load test > >> > > > > > > > > > > > > > (except > >> > delay > >> > > in > >> > > > > > > server > >> > > > > > > > > > > > > > response time for fetching files will > >> > > > > > > > > > > > > > impact the > >> > > > > javascript > >> > > > > > > > > > > > > > execution on client side, but that can be > >> > compensated > >> > > > w/ > >> > > > > > > JMeter > >> > > > > > > > > > > > > > load test against server with 1+ Selenium > >> > > > > > > > > > > > > > test > >> > > running > >> > > > at > >> > > > > > > same > >> > > > > > > > > > > > > > time to gauge client side performance of > >> > > > > > > > > > > > > > site/app > >> > in > >> > > > > > > browser). > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > If the javascript does execute AJAX > >> > > > > > > > > > > > > > requests, you > >> > > need > >> > > > to > >> > > > > > > > figure > >> > > > > > > > > > > > > > out the HTTP requests made and mimic that > >> > > > > > > > > > > > > > in JMeter > >> > > as > >> > > > > part > >> > > > > > > of > >> > > > > > > > > your > >> > > > > > > > > > > test. > >> > > > > > > > > > > > > > You can get that reading dev/design docs, > >> > > > > > > > > > > > > > or > >> > reverse > >> > > > > > > > > > > > > > engineer/traffic sniffing the app while > >> > > > > > > > > > > > > > doing > >> > manual > >> > > > > > testing. > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Fri, Feb 1, 2013 at 1:45 PM, Zippy > >> > > > > > > > > > > > > > Zeppoli <[email protected] > >> > > > > > > > > > > > > > >wrote: > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Hello, > >> > > > > > > > > > > > > > > If I have a website which requires > >> > > > > > > > > > > > > > > logging in, > >> > and > >> > > > > > > executing > >> > > > > > > > > > > > > > > complex javascript actions, how would I > >> > > > > > > > > > > > > > > do this > >> > (if > >> > > > at > >> > > > > > all) > >> > > > > > > > in > >> > > > > > > > > > jmeter? > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > I've heard of writing groovy scripts to > >> > > > > > > > > > > > > > > do this > >> > but > >> > > > > this > >> > > > > > > > sounds > >> > > > > > > > > > > > > > > like a > >> > > > > > > > > > > > > > lot > >> > > > > > > > > > > > > > > of work / maintenance. > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Thank you. > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > >> > > > --------------------------------------------------------------- > >> > > > ------ > >> > > > > > > > > > > > To unsubscribe, e-mail: > >> > > [email protected] > >> > > > > > > > > > > > For additional commands, e-mail: > >> > > > [email protected] > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > >> > ------------------------------------------------------------------- > >> > -- > >> > > > > > > > > > To unsubscribe, e-mail: > >> > > > > > > > > > [email protected] > >> > > > > > > > > > For additional commands, e-mail: > >> > [email protected] > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
