RE: Emulating a browsers resource download patterns
Thanks for the thoughts Deepak, after finding bigger issues with virtually every other software package I played with over the last few days I'm back. :) And Taking your advice on separating static content (not on a CDN in my case) from dynamic pages. It's an extra level of abstraction which makes it harder to emulate user actions, but your points are well taken and implemented now. One technical question: I set up a test as follows: Simple Controller Debug Sample Throughput Controller (50%) Half Time Debug Sample I set the iterations to LOOP FOREVER with 1 THREAD, so I would expect results something like: Debug Sample Half Time Debug Sample Debug Sample Debug Sample Half Time Debug Sample Debug Sample Debug Sample But what I get is the two interleaved for the first ~90 iterations, and then only 'Debug Sample' is executed after that. Debug Sample Half Time Debug Sample Debug Sample Half Time Debug Sample Debug Sample Half Time Debug Sample ...after ~90 such samples Debug Sample Debug Sample Debug Sample Debug Sample This doesn't jive with what I would expect after reading the documentation. -Original Message- From: Deepak Shetty [mailto:shet...@gmail.com] Sent: Sunday, October 09, 2011 11:52 AM To: JMeter Users List Subject: Re: Emulating a browsers resource download patterns it's not possible or worthwhile to try to replicate concurrent resource fetching. Did I imply that? - not intentional. I said depending on your situation a) it might not be worth it (but only you can know this based on your app) and there might be easier ways to get the answer you want(again application dependent). b) Such things are better measured(today) by tools that drive the browser. If there's no way to do this I, for one, am sure to put JMeter on the shelf and consider other options. If you want to know the exact time for a page to load - perhaps. If all you want to know is given a load , do 90% of my pages return under 3 seconds or given a functional / implementation change what impact to the site? then you might want to reconsider. I run most of my static file's separate to my dynamic files and guesstimate the entire page times (because Ive already verified that for the loads under consideration the static file times do not vary significantly whether i run them concurrently or sequentially) - the limiting factor for my pages are the dynamic pages which slow down far earlier than the static files. But then we have a sort of CDN and minify and gzip and cache and etags etc etc and do not have large static files. It works reasonably well(for me) - your mileage may vary. If you do this type of guesstimate it is also easy to provide an answer as to what happens when the user is located in Asia instead of the US for e.g. or if he has a full cache rather than an empty one. if Jmeter worked as you wanted you'd have to run the tests again with tweaked settings. Your approach of specifying URL's also has a maintenance problem of dealing with change. However yes the embedded resources is limiting and you should raise feature requests where you feel it lacks. Issue has already been discussed multiple times in the archives :). regards deepak On Sun, Oct 9, 2011 at 11:11 AM, David Parks davidpark...@yahoo.com wrote: Thanks for the response Deepak, but I will respectfully disagree with the assertion that states that it's not possible or worthwhile to try to replicate concurrent resource fetching. Of course each browser has differences, even different numbers of concurrent downloads, but JMeter is already attempting to emulate this in the HttpSampler with a concurrent download option. It would be easy to resolve the issue just by adding a set of URLs to the HttpSampler which are statically defined embedded resources (those it doesn't parse out we can just add ourselves), and have it download those using its X number of concurrent connections feature. I wouldn't try to solve the problem by improving how JMeter parses the resources out. Although it's always nice to improve that feature, JMeter can never be 100% for all applications. I have worked for 9 years as a consultant for HP's LoadRunner and BAC testing tools and will tell you even those million dollar testing packages can't parse all the resources from an HTML page (though they add static resources to a DL list automatically of course). Everything we do is going to be an approximation of reality, it's up to each application engineer to know their app and to do the best they can to approximate it. But having the ability to make a reasonable approximation is critical - and this is fundamental functionality used by all browsers in virtually every real-world scenario. To download each unparsed resource of a page in sequence makes it extremely difficult to built a semi-realistic test case, and the further we get from realistic the less valuable a tool like this becomes. If there's no way to do this I, for one, am sure
RE: Emulating a browsers resource download patterns
Ugg, please ignore this. Oblivious user mistake - I had it set to total iterations. -Original Message- From: David Parks [mailto:davidpark...@yahoo.com] Sent: Monday, October 10, 2011 11:30 AM To: 'JMeter Users List' Subject: RE: Emulating a browsers resource download patterns Thanks for the thoughts Deepak, after finding bigger issues with virtually every other software package I played with over the last few days I'm back. :) And Taking your advice on separating static content (not on a CDN in my case) from dynamic pages. It's an extra level of abstraction which makes it harder to emulate user actions, but your points are well taken and implemented now. One technical question: I set up a test as follows: Simple Controller Debug Sample Throughput Controller (50%) Half Time Debug Sample I set the iterations to LOOP FOREVER with 1 THREAD, so I would expect results something like: Debug Sample Half Time Debug Sample Debug Sample Debug Sample Half Time Debug Sample Debug Sample Debug Sample But what I get is the two interleaved for the first ~90 iterations, and then only 'Debug Sample' is executed after that. Debug Sample Half Time Debug Sample Debug Sample Half Time Debug Sample Debug Sample Half Time Debug Sample ...after ~90 such samples Debug Sample Debug Sample Debug Sample Debug Sample This doesn't jive with what I would expect after reading the documentation. -Original Message- From: Deepak Shetty [mailto:shet...@gmail.com] Sent: Sunday, October 09, 2011 11:52 AM To: JMeter Users List Subject: Re: Emulating a browsers resource download patterns it's not possible or worthwhile to try to replicate concurrent resource fetching. Did I imply that? - not intentional. I said depending on your situation a) it might not be worth it (but only you can know this based on your app) and there might be easier ways to get the answer you want(again application dependent). b) Such things are better measured(today) by tools that drive the browser. If there's no way to do this I, for one, am sure to put JMeter on the shelf and consider other options. If you want to know the exact time for a page to load - perhaps. If all you want to know is given a load , do 90% of my pages return under 3 seconds or given a functional / implementation change what impact to the site? then you might want to reconsider. I run most of my static file's separate to my dynamic files and guesstimate the entire page times (because Ive already verified that for the loads under consideration the static file times do not vary significantly whether i run them concurrently or sequentially) - the limiting factor for my pages are the dynamic pages which slow down far earlier than the static files. But then we have a sort of CDN and minify and gzip and cache and etags etc etc and do not have large static files. It works reasonably well(for me) - your mileage may vary. If you do this type of guesstimate it is also easy to provide an answer as to what happens when the user is located in Asia instead of the US for e.g. or if he has a full cache rather than an empty one. if Jmeter worked as you wanted you'd have to run the tests again with tweaked settings. Your approach of specifying URL's also has a maintenance problem of dealing with change. However yes the embedded resources is limiting and you should raise feature requests where you feel it lacks. Issue has already been discussed multiple times in the archives :). regards deepak On Sun, Oct 9, 2011 at 11:11 AM, David Parks davidpark...@yahoo.com wrote: Thanks for the response Deepak, but I will respectfully disagree with the assertion that states that it's not possible or worthwhile to try to replicate concurrent resource fetching. Of course each browser has differences, even different numbers of concurrent downloads, but JMeter is already attempting to emulate this in the HttpSampler with a concurrent download option. It would be easy to resolve the issue just by adding a set of URLs to the HttpSampler which are statically defined embedded resources (those it doesn't parse out we can just add ourselves), and have it download those using its X number of concurrent connections feature. I wouldn't try to solve the problem by improving how JMeter parses the resources out. Although it's always nice to improve that feature, JMeter can never be 100% for all applications. I have worked for 9 years as a consultant for HP's LoadRunner and BAC testing tools and will tell you even those million dollar testing packages can't parse all the resources from an HTML page (though they add static resources to a DL list automatically of course). Everything we do is going to be an approximation of reality, it's up to each application engineer to know their app and to do the best they can to approximate it. But having the ability to make a reasonable approximation is critical - and this is fundamental functionality used by all browsers
Re: Emulating a browsers resource download patterns
From the release note for 2.5.1 Additional known bugs: Version 2.5 introduced a concurrent download feature for embedded HTML resources. Unfortunately this may result in corrupted downloads or other errors (bugs 51918[1] and 51919[2]). We will fix these bugs as soon as possible; meanwhile the feature should not be used. But the HTTP Request Sampler only parses out about 1/5th of the resources on the page, There are some circumstances in which this is possible(other than bugs!) . e.g. Background Images referred to in CSS files (Jmeter will download resources for a page , but not resources within the resources like CSS files which refer to images). Dynamically added resources (from AJAX calls or javascript) wont work either - As always JMeter is not a browser. You might need to put in a feature request in bugzilla if you can identify patterns within your page that the embedded resource parser isnt picking up. The problem with the type of results you want actually depend on multiple questions - Which browser? Which setting on the browser? (e.g. IE which controls how resources are cached)? When is a page considered loaded - especially when you account for DHTML rich/AJAX applications? And is it worth it (for e.g. if you use a CDN it probably isnt). And if you still really want to know its probably to use browser driven tools like selenium (or selenium grid) to find out the answer - or selenium + jmeter where jmeter generates the load you want and a single selenium instance can browse the site and record page times ). Using JMeter would need you to estimate this answer (for e.g. if you get values for each individual resources then using firebug and the network tab you can figure out the time for page load). regards deepak On Sun, Oct 9, 2011 at 9:34 AM, David Parks davidpark...@yahoo.com wrote: A browser typically opens about 4 connections to download all of the resources of a page. I'd naturally like to emulate this behavior in my test cases. I see that the HTTP Request Sampler has an option to do exactly this with the parsed Embedded Resources. But the HTTP Request Sampler only parses out about 1/5th of the resources on the page, instead I just record the page and all its resources with a Recording Controller. But now each resource is an HTTP Sampler its self and I have no way of emulating the 4 concurrent downloads, they download in sequence. Thus I don't see how to even come close to accurately simulating browser load, and certainly don't see a way to accurately time a page download. Perhaps I missed something? But I didn't see this question addressed in the docs or FAQ's. Thanks, Dave - To unsubscribe, e-mail: jmeter-user-unsubscr...@jakarta.apache.org For additional commands, e-mail: jmeter-user-h...@jakarta.apache.org
RE: Emulating a browsers resource download patterns
Thanks for the response Deepak, but I will respectfully disagree with the assertion that states that it's not possible or worthwhile to try to replicate concurrent resource fetching. Of course each browser has differences, even different numbers of concurrent downloads, but JMeter is already attempting to emulate this in the HttpSampler with a concurrent download option. It would be easy to resolve the issue just by adding a set of URLs to the HttpSampler which are statically defined embedded resources (those it doesn't parse out we can just add ourselves), and have it download those using its X number of concurrent connections feature. I wouldn't try to solve the problem by improving how JMeter parses the resources out. Although it's always nice to improve that feature, JMeter can never be 100% for all applications. I have worked for 9 years as a consultant for HP's LoadRunner and BAC testing tools and will tell you even those million dollar testing packages can't parse all the resources from an HTML page (though they add static resources to a DL list automatically of course). Everything we do is going to be an approximation of reality, it's up to each application engineer to know their app and to do the best they can to approximate it. But having the ability to make a reasonable approximation is critical - and this is fundamental functionality used by all browsers in virtually every real-world scenario. To download each unparsed resource of a page in sequence makes it extremely difficult to built a semi-realistic test case, and the further we get from realistic the less valuable a tool like this becomes. If there's no way to do this I, for one, am sure to put JMeter on the shelf and consider other options. But I'll be very surprised if this issue hasn't been discussed at infinitum already. -Original Message- From: Deepak Shetty [mailto:shet...@gmail.com] Sent: Sunday, October 09, 2011 10:30 AM To: JMeter Users List Subject: Re: Emulating a browsers resource download patterns From the release note for 2.5.1 Additional known bugs: Version 2.5 introduced a concurrent download feature for embedded HTML resources. Unfortunately this may result in corrupted downloads or other errors (bugs 51918[1] and 51919[2]). We will fix these bugs as soon as possible; meanwhile the feature should not be used. But the HTTP Request Sampler only parses out about 1/5th of the resources on the page, There are some circumstances in which this is possible(other than bugs!) . e.g. Background Images referred to in CSS files (Jmeter will download resources for a page , but not resources within the resources like CSS files which refer to images). Dynamically added resources (from AJAX calls or javascript) wont work either - As always JMeter is not a browser. You might need to put in a feature request in bugzilla if you can identify patterns within your page that the embedded resource parser isnt picking up. The problem with the type of results you want actually depend on multiple questions - Which browser? Which setting on the browser? (e.g. IE which controls how resources are cached)? When is a page considered loaded - especially when you account for DHTML rich/AJAX applications? And is it worth it (for e.g. if you use a CDN it probably isnt). And if you still really want to know its probably to use browser driven tools like selenium (or selenium grid) to find out the answer - or selenium + jmeter where jmeter generates the load you want and a single selenium instance can browse the site and record page times ). Using JMeter would need you to estimate this answer (for e.g. if you get values for each individual resources then using firebug and the network tab you can figure out the time for page load). regards deepak On Sun, Oct 9, 2011 at 9:34 AM, David Parks davidpark...@yahoo.com wrote: A browser typically opens about 4 connections to download all of the resources of a page. I'd naturally like to emulate this behavior in my test cases. I see that the HTTP Request Sampler has an option to do exactly this with the parsed Embedded Resources. But the HTTP Request Sampler only parses out about 1/5th of the resources on the page, instead I just record the page and all its resources with a Recording Controller. But now each resource is an HTTP Sampler its self and I have no way of emulating the 4 concurrent downloads, they download in sequence. Thus I don't see how to even come close to accurately simulating browser load, and certainly don't see a way to accurately time a page download. Perhaps I missed something? But I didn't see this question addressed in the docs or FAQ's. Thanks, Dave - To unsubscribe, e-mail: jmeter-user-unsubscr...@jakarta.apache.org For additional commands, e-mail: jmeter-user-h...@jakarta.apache.org - To unsubscribe
Re: Emulating a browsers resource download patterns
it's not possible or worthwhile to try to replicate concurrent resource fetching. Did I imply that? - not intentional. I said depending on your situation a) it might not be worth it (but only you can know this based on your app) and there might be easier ways to get the answer you want(again application dependent). b) Such things are better measured(today) by tools that drive the browser. If there's no way to do this I, for one, am sure to put JMeter on the shelf and consider other options. If you want to know the exact time for a page to load - perhaps. If all you want to know is given a load , do 90% of my pages return under 3 seconds or given a functional / implementation change what impact to the site? then you might want to reconsider. I run most of my static file's separate to my dynamic files and guesstimate the entire page times (because Ive already verified that for the loads under consideration the static file times do not vary significantly whether i run them concurrently or sequentially) - the limiting factor for my pages are the dynamic pages which slow down far earlier than the static files. But then we have a sort of CDN and minify and gzip and cache and etags etc etc and do not have large static files. It works reasonably well(for me) - your mileage may vary. If you do this type of guesstimate it is also easy to provide an answer as to what happens when the user is located in Asia instead of the US for e.g. or if he has a full cache rather than an empty one. if Jmeter worked as you wanted you'd have to run the tests again with tweaked settings. Your approach of specifying URL's also has a maintenance problem of dealing with change. However yes the embedded resources is limiting and you should raise feature requests where you feel it lacks. Issue has already been discussed multiple times in the archives :). regards deepak On Sun, Oct 9, 2011 at 11:11 AM, David Parks davidpark...@yahoo.com wrote: Thanks for the response Deepak, but I will respectfully disagree with the assertion that states that it's not possible or worthwhile to try to replicate concurrent resource fetching. Of course each browser has differences, even different numbers of concurrent downloads, but JMeter is already attempting to emulate this in the HttpSampler with a concurrent download option. It would be easy to resolve the issue just by adding a set of URLs to the HttpSampler which are statically defined embedded resources (those it doesn't parse out we can just add ourselves), and have it download those using its X number of concurrent connections feature. I wouldn't try to solve the problem by improving how JMeter parses the resources out. Although it's always nice to improve that feature, JMeter can never be 100% for all applications. I have worked for 9 years as a consultant for HP's LoadRunner and BAC testing tools and will tell you even those million dollar testing packages can't parse all the resources from an HTML page (though they add static resources to a DL list automatically of course). Everything we do is going to be an approximation of reality, it's up to each application engineer to know their app and to do the best they can to approximate it. But having the ability to make a reasonable approximation is critical - and this is fundamental functionality used by all browsers in virtually every real-world scenario. To download each unparsed resource of a page in sequence makes it extremely difficult to built a semi-realistic test case, and the further we get from realistic the less valuable a tool like this becomes. If there's no way to do this I, for one, am sure to put JMeter on the shelf and consider other options. But I'll be very surprised if this issue hasn't been discussed at infinitum already. -Original Message- From: Deepak Shetty [mailto:shet...@gmail.com] Sent: Sunday, October 09, 2011 10:30 AM To: JMeter Users List Subject: Re: Emulating a browsers resource download patterns From the release note for 2.5.1 Additional known bugs: Version 2.5 introduced a concurrent download feature for embedded HTML resources. Unfortunately this may result in corrupted downloads or other errors (bugs 51918[1] and 51919[2]). We will fix these bugs as soon as possible; meanwhile the feature should not be used. But the HTTP Request Sampler only parses out about 1/5th of the resources on the page, There are some circumstances in which this is possible(other than bugs!) . e.g. Background Images referred to in CSS files (Jmeter will download resources for a page , but not resources within the resources like CSS files which refer to images). Dynamically added resources (from AJAX calls or javascript) wont work either - As always JMeter is not a browser. You might need to put in a feature request in bugzilla if you can identify patterns within your page that the embedded resource parser isnt picking up. The problem with the type of results
