There seems to be a big misunderstanding of what is the benefit of using a browser. I think this thread at this point is far off topic, although it looks like a lot of good knowledge was shaken out of it, so I will consider it time well spent, and a question well asked.
And now the clarification: Comparing RUM & Synthetic Page Load Times November 14, 2012 5:30 pm | 13 Comments Yesterday I read Etsy’s October 2012 Site Performance Report. Etsy is one of only a handful of companies that publish their performance stats with explanations and future plans. It’s really valuable (and brave!), and gives other developers an opportunity to learn from an industry leader. In this article Etsy mentions that the page load time stats are gathered from a private instance of WebPagetest. They explain their use of synthetically-generated measurements instead of RUM (Real User Monitoring) data: You might be surprised that we are using synthetic tests for this front-end report instead of Real User Monitoring (RUM) data. RUM is a big part of performance monitoring at Etsy, but when we are looking at trends in front-end performance over time, synthetic testing allows us to eliminate much of the network variability that is inherent in real user data. This helps us tie performance regressions to specific code changes, and get a more stable view of performance overall. Etsy’s choice of synthetic data for tracking performance as part of their automated build process totally makes sense. I’ve talked to many companies that do the same thing. Teams dealing with builds and code regressions should definitely do this. BUT… it’s important to include RUM data when sharing performance measurements beyond the internal devops team. Why should RUM data always be used when talking beyond the core team? The issue with only showing synthetic data is that it typically makes a website appear much faster than it actually is. This has been true since I first started tracking real user metrics back in 2004. My rule-of-thumb is that your real users are experiencing page load times that are twice as long as their corresponding synthetic measurements. >From >http://www.stevesouders.com/blog/2012/11/14/comparing-rum-synthetic-page-load-times/ On Wed, Feb 6, 2013 at 4:30 PM, Zippy Zeppoli <[email protected]> wrote: > > I think you may be missing the point. > Real load cannot be tested via HTTP interactions. > There is no DOM rendering happening. > I can make HTTP requests all day and it won't reflect the true response time > unless it's done through a browser. > > Recording a script in Jmeter proxy is trivial. Simulating *real* user load is > not it requires a browser and interactions with a web application. > > > On Tue, Feb 5, 2013 at 6:51 PM, Deepak Shetty <[email protected]> wrote: >> >> >Actually that does matter it cannot do JavaScript. If a request requires >> >you need to be able to click a JavaScript button then the request will >> >never happen. >> The point is that what happens when the button is clicked? Assuming its a >> server - ajax call then A HTTP call is made and some parameters are passed >> and some values are returned. Thats whats important for the load test , not >> the fact that javascript was executed. >> So when you record the script , you will be the person clicking the >> button(you are recording your actions) , JMeter will record every >> interaction that makes a call to the server and will record this as a >> separate HTTP request and when you run the script the same request will be >> made as if someone clicked the button! >> >> You dont need to use the recorder either , you can modify the script >> yourself. >> >> If the javascript didnt actually make any server side call - then it doesnt >> matter because you dont want to load test this anyway. >> >> Have you actually tried this? It sounds as if you have a problem recording >> your script and you probably have concluded that JMeter doesnt do >> javascript (true) and hence cant test websites that do javascript/ajax >> (false) >> >> >Real browser is needed >> Not for a good deal of use cases - as many of the people on this mailing >> list can attest too. >> >> >> >> >> >> On Tue, Feb 5, 2013 at 6:44 PM, Zippy Zeppoli <[email protected]>wrote: >> >> > Deepak, >> > Actually that does matter it cannot do JavaScript. If a request requires >> > you need to be able to click a JavaScript button then the request will >> > never happen. No request will ever be made. Also testing true web >> > performance requires rendering the DOM, not just initiating HTTP requests >> > and recording the response time, rps, etc. >> > >> > Real browser is needed, with JavaScript, and Jmeter doesn't integrate well >> > with this, it isn't designed for this, which is understandable. The problem >> > is there is a gap between real browser testing (owned by third party >> > companies) and open source tools (Jmeter). There's nothing in between for >> > real-browser based performance testing. I could go into why, but its off >> > topic of this list, and I'd rather spare everyone the gas. >> > >> > Point being, Jmeter cannot solve my problem, without some serious >> > customization. >> > >> > >> > >> > On Tue, Feb 5, 2013 at 6:33 PM, Deepak Shetty <[email protected]> wrote: >> > >> > > Hi >> > > You are getting too caught up in the JMeter doesnt do javascript thing. >> > In >> > > most cases it doesnt matter. >> > > You have a webserver that is receiving HTTP requests - whether those >> > > requests are generated via the user clicking a link or via AJAX or via >> > > flash is hardly relevant to the webserver. It sees HTTP requests and >> > sends >> > > HTTP responses. >> > > JMeter deals with HTTP request and responses. As long as you can make the >> > > same request that your javascript is making (which you can see via the >> > > recording feature) , you can test it with Jmeter. The fact that the proxy >> > > cant record javascript is irrelevant - it only needs to record the >> > > requests. If your javascript is responsible for generating the requests >> > in >> > > some way , then you have to replicate that within JMeter. >> > > >> > > Some caveats here are >> > > a. Your toolkit should be good when you need to parameterise(JQuery is >> > > great , GWT sucks) >> > > b. If you need to perform some AJAX requests in parallel then JMeter isnt >> > > quite there yet. >> > > >> > > Its probably true that it is easier writing scripts for Selenium like >> > tools >> > > - but they also need more resources as well as given how often a UI >> > changes >> > > as opposed to the HTTP request/response pair , sometimes JMeter scripts >> > are >> > > more resilient to change. >> > > >> > > regards >> > > deepak >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > On Tue, Feb 5, 2013 at 5:29 PM, Zippy Zeppoli <[email protected] >> > > >wrote: >> > > >> > > > Proxy won't work for clicking on JavaScript. >> > > > Selenium will, however, Selenium isn't designed for performance >> > testing, >> > > > with the exception of BrowserMob, who has perfected it. >> > > > >> > > > If JMeter had some kind of (good) Selenium integration then it might be >> > > > able to achieve it. >> > > > >> > > > I think JMeter is a good tool, but it's not built for this kind of >> > work. >> > > > >> > > > On Tue, Feb 5, 2013 at 5:20 PM, David Luu <[email protected]> wrote: >> > > > >> > > > > I'm not familiar with the full feature set of BrowserMob, but JMeter >> > > can >> > > > do >> > > > > what you want, easy or hard, depending on your needs. >> > > > > >> > > > > If all you need is simple record & playback, JMeter proxy is best >> > > option. >> > > > > Just record w/ proxy, it generates a basic test plan that you can >> > save >> > > to >> > > > > file and run w/ X threads to generate load, etc. >> > > > > >> > > > > But that approach will always use the same user, same data in the >> > AJAX >> > > > > calls during load generation. >> > > > > >> > > > > If you need them to be unique & parameterized, that's where it takes >> > > some >> > > > > work to go in and cleanup the proxy recording w/ paramaterized data, >> > > > > response checking (if more than just checking HTTP 200 OKs). And >> > you'll >> > > > > find proxy recording generates a lot of unnecessary HTTP requests for >> > > > HTML, >> > > > > CSS, image files that aren't necessary of interest as you're testing >> > > the >> > > > > AJAX calls, which is where you filter out only the requests of >> > interest >> > > > and >> > > > > only use those in test plan, etc. >> > > > > >> > > > > I believe this kind of work is needed regardless of what tool you >> > use, >> > > > some >> > > > > tools just make it easier, but there's still work to do. Not just a >> > one >> > > > > touch button approach unless all you need is simple record & >> > playback. >> > > > > >> > > > > On Tue, Feb 5, 2013 at 5:08 PM, Zippy Zeppoli < >> > [email protected] >> > > > > >wrote: >> > > > > >> > > > > > Will probably just buy BrowserMob. >> > > > > > Too bad there isn't an open source framework to already do this. >> > > > > > >> > > > > > Building this is a yak shave, and I need to be testing, not >> > building >> > > a >> > > > > test >> > > > > > harness thats probably going to break on the next release. >> > > > > > >> > > > > > On Tue, Feb 5, 2013 at 3:40 PM, David Luu <[email protected]> >> > > wrote: >> > > > > > >> > > > > > > JMeter proxy is the most integrated approach, but I find it gives >> > > > "too >> > > > > > much >> > > > > > > information", so I tend to use external tools like browser >> > traffic >> > > > > > sniffers >> > > > > > > (HttpFox, livehttpheaders, ieHttpHeaders) to see what HTTP >> > requests >> > > > are >> > > > > > > made for AJAX calls for just the requests & responses I'm >> > > interested >> > > > > in, >> > > > > > > easier to manage. But that's just my opinion, others might find >> > the >> > > > > > JMeter >> > > > > > > proxy easier to use. >> > > > > > > >> > > > > > > Whatever approach you take, you just need to know what HTTP >> > > requests >> > > > > are >> > > > > > > made by the AJAX calls to replicate in JMeter, and parameterize >> > > those >> > > > > > > requests to take in dynamic/test data as needed, assert >> > appropriate >> > > > > > > response data, etc. In this case, WebDriver can be dropped from >> > the >> > > > > > > equation, just gives more overhead and lowers scalability in load >> > > > > > > generation. >> > > > > > > >> > > > > > > On Tue, Feb 5, 2013 at 1:55 PM, Philippe Mouawad < >> > > > > > > [email protected]> wrote: >> > > > > > > >> > > > > > > > AjaxCall will emit a network call which JMeter will capture. >> > > > > > > > >> > > > > > > > Use JMeter Server Proxy and you should have the Ajax Calls >> > > > recorded. >> > > > > > > > >> > > > > > > > Regards >> > > > > > > > Philippe M. >> > > > > > > > >> > > > > > > > Follow me on twitter <https://twitter.com/philmdot> >> > > > > > > > >> > > > > > > > UBIK-INGENIERIE on TWITTER <https://twitter.com/ubikingenierie >> > > >> > > > > > > > >> > > > > > > > UBIK LOAD PACK BLOG <http://www.ubik-ingenierie.com/blog/> >> > > > > > > > >> > > > > > > > On Tue, Feb 5, 2013 at 10:52 PM, Zippy Zeppoli < >> > > > > [email protected] >> > > > > > > > >wrote: >> > > > > > > > >> > > > > > > > > To clarify: >> > > > > > > > > An example would be: >> > > > > > > > > 1) log in via a form post >> > > > > > > > > 2) look at orders in an ecommerce interface (AJAX call) >> > > > > > > > > 3) click on result to view order detail (AJAX) >> > > > > > > > > >> > > > > > > > > Pretty much a standard ecommerce transaction, but the >> > interface >> > > > is >> > > > > > all >> > > > > > > > > javascript. >> > > > > > > > > >> > > > > > > > > On Mon, Feb 4, 2013 at 7:40 PM, Stott, Charlie < >> > > [email protected]> >> > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > > -----Original Message----- >> > > > > > > > > > > From: David Luu [mailto:[email protected]] >> > > > > > > > > > > Sent: Tuesday, 5 February 2013 1:18 PM >> > > > > > > > > > > To: JMeter Users List >> > > > > > > > > > > Subject: Re: complex javascript actions in jmeter load >> > test >> > > > > > > > > > > >> > > > > > > > > > > >> You can use webdriver from jmeter. Create a >> > webdriver >> > > > > class >> > > > > > > that >> > > > > > > > > > > performs the requests and runs the javascript via the >> > > > browser, >> > > > > > then >> > > > > > > > > > run/call >> > > > > > > > > > > it from a BSF or JSR sampler. >> > > > > > > > > > > >> > > > > > > > > > > Just to clarify, I take it that's only worthwhile to do >> > (in >> > > > > terms >> > > > > > > of >> > > > > > > > > > > scalability) when using PhantomJSDriver or HtmlUnitDriver >> > > or >> > > > > > > > > > FirefoxDriver >> > > > > > > > > > > (on Linux with xvfb) with JMeter this way? >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > "Worthwhile" depends on assumptions. >> > > > > > > > > > >> > > > > > > > > > > Because otherwise, the browser GUI is the scalability >> > > > limiting >> > > > > > > factor >> > > > > > > > > > even >> > > > > > > > > > > with JMeter and Grid deployment, and in that case, no >> > > > > difference >> > > > > > in >> > > > > > > > > using >> > > > > > > > > > > WebDriver outside JMeter to do performance tests except >> > if >> > > > one >> > > > > > > wants >> > > > > > > > > the >> > > > > > > > > > > JMeter logging/reporting facilities to help performance >> > > test, >> > > > > > > because >> > > > > > > > > > there's >> > > > > > > > > > > no or minimal scalabiity gain. >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > Seems the poster is more at the capability/viability stage >> > of >> > > > > > > > developing >> > > > > > > > > > tests? We would need much more information to start >> > advising >> > > > on >> > > > > > the >> > > > > > > > > entire >> > > > > > > > > > load testing process, start to finish, and what challenges >> > > may >> > > > > face >> > > > > > > the >> > > > > > > > > > poster along the way. >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > On Mon, Feb 4, 2013 at 4:42 PM, Stott, Charlie < >> > > > > [email protected]> >> > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > You can use webdriver from jmeter. Create a webdriver >> > > > class >> > > > > > that >> > > > > > > > > > > > performs the requests and runs the javascript via the >> > > > > browser, >> > > > > > > then >> > > > > > > > > > > > run/call it from a BSF or JSR sampler. >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > -----Original Message----- >> > > > > > > > > > > > > From: Zippy Zeppoli [mailto:[email protected]] >> > > > > > > > > > > > > Sent: Tuesday, 5 February 2013 9:28 AM >> > > > > > > > > > > > > To: JMeter Users List >> > > > > > > > > > > > > Subject: Re: complex javascript actions in jmeter >> > load >> > > > test >> > > > > > > > > > > > > >> > > > > > > > > > > > > The problem is Selenium has no performance testing >> > > > harness. >> > > > > > > > > > > > > Sucks that it seems BrowserMob (paid solution) is the >> > > > only >> > > > > > > solid >> > > > > > > > > > option. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Until someone builds something with Phantom.js, but >> > it >> > > > > seems >> > > > > > > > JMeter >> > > > > > > > > > > > > isn't going to cut it here. >> > > > > > > > > > > > > >> > > > > > > > > > > > > On Fri, Feb 1, 2013 at 5:40 PM, David Luu < >> > > > > > [email protected]> >> > > > > > > > > > wrote: >> > > > > > > > > > > > > >> > > > > > > > > > > > > > You'll need to figure out what the complex >> > javascript >> > > > > does. >> > > > > > > > Does >> > > > > > > > > > > > > > it make any AJAX requests, or is it all local >> > client >> > > > side >> > > > > > > > > > > > processing/rendering? >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > If it's all local, then there's no point testing it >> > > > with >> > > > > > > > JMeter, >> > > > > > > > > > > > > > that's client side browser testing better done with >> > > > > > Selenium. >> > > > > > > > It >> > > > > > > > > > > > > > won't impact the server side load test (except >> > delay >> > > in >> > > > > > > server >> > > > > > > > > > > > > > response time for fetching files will impact the >> > > > > javascript >> > > > > > > > > > > > > > execution on client side, but that can be >> > compensated >> > > > w/ >> > > > > > > JMeter >> > > > > > > > > > > > > > load test against server with 1+ Selenium test >> > > running >> > > > at >> > > > > > > same >> > > > > > > > > > > > > > time to gauge client side performance of site/app >> > in >> > > > > > > browser). >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > If the javascript does execute AJAX requests, you >> > > need >> > > > to >> > > > > > > > figure >> > > > > > > > > > > > > > out the HTTP requests made and mimic that in JMeter >> > > as >> > > > > part >> > > > > > > of >> > > > > > > > > your >> > > > > > > > > > > test. >> > > > > > > > > > > > > > You can get that reading dev/design docs, or >> > reverse >> > > > > > > > > > > > > > engineer/traffic sniffing the app while doing >> > manual >> > > > > > testing. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Fri, Feb 1, 2013 at 1:45 PM, Zippy Zeppoli >> > > > > > > > > > > > > > <[email protected] >> > > > > > > > > > > > > > >wrote: >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Hello, >> > > > > > > > > > > > > > > If I have a website which requires logging in, >> > and >> > > > > > > executing >> > > > > > > > > > > > > > > complex javascript actions, how would I do this >> > (if >> > > > at >> > > > > > all) >> > > > > > > > in >> > > > > > > > > > jmeter? >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > I've heard of writing groovy scripts to do this >> > but >> > > > > this >> > > > > > > > sounds >> > > > > > > > > > > > > > > like a >> > > > > > > > > > > > > > lot >> > > > > > > > > > > > > > > of work / maintenance. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Thank you. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > >> > > > --------------------------------------------------------------------- >> > > > > > > > > > > > To unsubscribe, e-mail: >> > > [email protected] >> > > > > > > > > > > > For additional commands, e-mail: >> > > > [email protected] >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > >> > --------------------------------------------------------------------- >> > > > > > > > > > To unsubscribe, e-mail: [email protected] >> > > > > > > > > > For additional commands, e-mail: >> > [email protected] >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
