I agree that jmeter rarely performs poorly in non-GUI mode, unless you allocate to much memory to it.
On Thu, Dec 15, 2011 at 1:12 PM, Adrian Speteanu <[email protected]>wrote: > In my experience, "stange pauses" usually points to the application. > > Of course, you have to make sure its not you (overlook in the test script, > jmeter's configuration), or the test client environment (OS, network). BUT > after all that is exhausted is usually the application or one of the layers > of the system under test (like the database, usually more trivial stuff > like a small connection pool to the database, or GC of the application > under test). > > If you suspect its GC, divide the load you generate with the script in 3, > and instead of testing with one jmeter instance, use 3 (if you can test > like this on 3 different machines - the better). Also allocate less memory > to jmeter now. What's the probability that all instances get paused by GC > at the same time? Also, if you allocate less memory, stop-the-world GC > takes less time. If you still have response times pauses in the same > intervals - it has to be the system under test. Note: > > On Thu, Dec 15, 2011 at 3:51 AM, Robin D. Wilson <[email protected]>wrote: > >> Yes it is. I can turn it off if you think that matters. >> >> -- >> Robin D. Wilson >> >> >> >> On Dec 14, 2011, at 6:42 PM, [email protected] wrote: >> >> > I'm curious to know if the system under test is https >> > >> > >> > -----Original Message----- >> > From: sebb <[email protected]> >> > Date: Thu, 15 Dec 2011 00:35:39 >> > To: JMeter Users List<[email protected]> >> > Reply-To: "JMeter Users List" <[email protected]> >> > Subject: Re: Strange 'pause' activity during testing >> > >> > On 15 December 2011 00:22, Robin D. Wilson <[email protected]> wrote: >> >> I can use non-GUI mode for testing out the problem, but with the exact >> same >> >> test - just switching the 'Retrieve All Embedded...' disabled, the >> problem >> >> goes away. >> >> >> >> The only listeners I have enabled are the summary listener - the >> PerfMon >> >> listeners (1 for each server) - and a 'tree' listener, but it only >> reports >> >> 'errors'. >> >> >> >> I can run the test with 'Retrieve All Embedded...' enabled, I see the >> >> problem every time. >> > >> > Maybe then there is a problem with the scanning of the HTML to extract >> > the embedded resources, or maybe one of the embedded resources is a >> > tar-pit. >> > >> > Do all the expected embedded resources get downloaded? >> > >> > Are there any unusually long elapsed times for embedded resources? >> > >> > Or large gaps between the parent sample download completion and the >> > start of the first embedded resource? >> > >> > That would suggest the page took a while to parse. >> > >> > You'll need to select the optiion to "save subresults" in order to see >> > the embedded samples. >> > >> >> If I just uncheck the box in the Request Defaults and re-run the test, >> it >> >> never happens... So are you suggesting that I've hit some arbitrary >> limit of >> >> the GUI mode? >> > >> > No, just that GUI mode is inherently more resource intensive. >> > >> > BTW, you wrote you were running JMeter 2.4.1 - that does not exist, so >> > perhaps you meant the current version, 2.5.1? >> > >> > >> >> What I'm wondering is why: >> >> >> >> 1) It is so consistent - the 'stop' happens at nearly the exact same >> point >> >> in every test run, and for nearly the exact same duration - and it >> happens >> >> multiple times during the test run, in the same pattern. >> >> >> >> 2) I can increase the load 10X (even more if I want), but without the >> >> 'Retrieve All Embedded...' enabled, and it doesn’t happen. >> >> >> >> If it were resource related on the JMeter box, I would guess that >> changing >> >> the JVM memory config would show significant changes in behavior (like >> it >> >> would take longer to encounter the first 'stop'). If it were resource >> >> related on the servers, I would expect to see some sort of contention >> on the >> >> servers. >> >> >> >> Tomorrow I will try the thread dump stuff - to see if there is anything >> >> obvious in that output. >> >> >> >> From an "external" view - it looks like JMeter has some sort of 'pause' >> >> built into it. Since all machines involved are nearly completely >> 'idle' when >> >> the stop happens - it seems unlikely that I've run out of resources. >> The >> >> other thought is that there is a deadlock somewhere - and something >> has to >> >> timeout before it can free up the deadlock and start again. But if >> that were >> >> the case, I would expect it to happen at slightly different intervals >> in >> >> each test run - and not repeat in the same consistent pattern during >> the >> >> test run. >> >> >> >> Some thoughts on what I think I've ruled out: >> >> >> >> 1) If it were the network 'backing off' I would expect the PerfMon >> listener >> >> to stop updating during the 'stop' - but it trundles happily along. >> >> 2) If it were JVM garbage collections - I would have expected to see >> them >> >> running (or at least a 'stop-the-world' gc right before/after the >> 'stop' >> >> period) during the 'stop' period - but there are no gc events at all >> during >> >> the 'stop' period. >> >> 3) If it were WinXP resources, I would have expected the PerfMon >> listener to >> >> show some activity during the 'stop' period on the WinXP (JMeter >> client) box >> >> - nothing like that appears. >> >> >> >> I'll look at the thread dumps tomorrow and get back to you guys. >> Thanks for >> >> helping me ponder this. >> >> >> >> -- >> >> Robin D. Wilson >> >> Sr. Director of Web Development >> >> KingsIsle Entertainment, Inc. >> >> VOICE: 512-777-1861 >> >> www.KingsIsle.com >> >> >> >> >> >> -----Original Message----- >> >> From: sebb [mailto:[email protected]] >> >> Sent: Wednesday, December 14, 2011 5:27 PM >> >> To: JMeter Users List >> >> Subject: Re: Strange 'pause' activity during testing >> >> >> >> On 14 December 2011 22:20, Deepak Shetty <[email protected]> wrote: >> >>> I think the original post mentioned that GC was not running. >> >>> >> >>> can you take a thread dump when there is a pause and see what the >> >>> threads are waiting on? >> >>> >> >> >> >> I would also try running JMeter in non-GUI mode; GUI mode is very >> expensive >> >> in resources. >> >> >> >> Ensure all non-essential Listeners are disabled, see: >> >> >> >> http://jmeter.apache.org/usermanual/best-practices.html#lean_mean >> >> >> >>> >> >>> On Wed, Dec 14, 2011 at 1:56 PM, Kirk <[email protected]> >> wrote: >> >>> >> >>>> might be GC.. me be that JMeter's threads are all hung up in your >> >>>> server doing stuff. >> >>>> >> >>>> Regards, >> >>>> Kirk >> >>>> On Dec 14, 2011, at 10:35 PM, Robin D. Wilson wrote: >> >>>> >> >>>>> I have a marginally complicated test case that performs a >> >> 'registration' >> >>>> on >> >>>>> my site. It gets the home page, POSTS a form, gets the response, >> >>>>> POSTS another form, gets that response, and then completes. The >> >>>>> test runs fine with 100 threads, and 30000 iterations - IF I don't >> >>>>> "Retrieve All >> >>>> Embedded >> >>>>> Resources from HTML Files". In this mode, I am really testing the >> >>>> throughput >> >>>>> of my 'tomcat' application, not the other elements of my system. >> >>>>> (I'm assuming that the other elements are being retrieved from our >> >>>>> Content >> >>>> Data >> >>>>> Network instead of our main system in this case.) >> >>>>> >> >>>>> If I enable the "Retrieve All Embedded Resources from HTML Files" >> >>>>> flag, >> >>>> and >> >>>>> tune the test down to 10 threads with 3000 total iterations, I >> >>>>> notice a >> >>>> very >> >>>>> strange behavior. The test runs along at a pretty good clip for the >> >>>>> first >> >>>>> ~600 iterations (about 1 min, 25 seconds into the run), and then it >> >>>>> just stops making requests for about 35 seconds. Then, it picks >> >>>>> back up again >> >>>> and >> >>>>> runs for another 1 m 25s, and then stops again for 35 seconds... >> (NOTE: >> >>>> with >> >>>>> the 10 threads, 3000 total iterations - but with "Retrieve All >> >>>> Embedded..." >> >>>>> disabled - I don't see the 'stop' behavior either - so it isn't >> >>>>> caused by tuning it down...) >> >>>>> >> >>>>> I recently added the "Perfmon Metrics Collector" to the test >> >>>>> script, so I could see if one of the servers was maxed out - but it >> >>>>> looks like all the servers are idle during the 'stop' period. >> >>>>> Likewise, I added the Perfmon >> >>>> for >> >>>>> the "localhost" (running the JMeter test) to see if it was swamped >> >>>>> - but >> >>>> it >> >>>>> too is idle during the 'stop'. I swapped out our network switch >> >>>>> (the test environment is on an isolated network switch) with a >> >>>>> _much_ higher >> >>>> capacity >> >>>>> switch - in case there was a network issue, still no change. >> >>>>> >> >>>>> I'm running out of ideas for things to check - so I thought I'd ask >> >>>>> you >> >>>> guys >> >>>>> if you have any suggestions of things I should look at. >> >>>>> >> >>>>> My system consists of: >> >>>>> >> >>>>> WinXP - running JMeter 2.4.1 - driving the test script in GUI >> >>>>> mode >> >>>>> Server 1 - running Red Hat Linux, with "Apache (2.2.21)" as >> >>>>> the web server - using AJP Proxy to Server 2 >> >>>>> Server 2 - running Red Hat Linux, with Tomcat 7.0.21 as the >> >>>>> App Server - connecting through Hibernate to Server 3 >> >>>>> Server 3 - running Red Hat Linux with MySQL 5.x as the DB >> >>>>> Server >> >>>>> >> >>>>> All 4 machines are running on a private switched network (32Gbs >> >>>> backplane). >> >>>>> >> >>>>> The requests are downloading about 3MB total (per thread per >> >>>>> iteration) >> >>>> over >> >>>>> 4 main URL requests, and 30+ 'Retrieve All Embedded' requests. >> >>>>> >> >>>>> At first I thought it was the network - but the new switch seemed >> >>>>> to deny that thought (the old switch had a much slower backplane). >> >>>>> Also, I'm >> >>>> having >> >>>>> no trouble collecting the PerfMon data during the 'stop' period - >> >>>>> so the network is still functioning just fine... >> >>>>> Then I thought it might be garbage collection on the tomcat - but I >> >>>> watched >> >>>>> the gc.log - and it doesn't do any GCs during the 'stop' period. >> >>>>> Then I thought it might be the garbage collection on the JMeter >> >>>>> side, so >> >>>> I >> >>>>> started the JMeter.bat from a 'cmd' prompt with gc logging enabled >> >>>>> - it doesn't do any GCs during the 'stop' period either. >> >>>>> >> >>>>> The apache, tomcat, and DB are all 'idle' (no CPU to speak of, no >> >>>>> network I/O, no disk I/O, etc.) during the 'stop' period. >> >>>>> >> >>>>> The JMeter box (WinXP) is doing very little during that time too (I >> >>>>> attribute the little bit of activity to displaying the PerfMon >> >>>>> graphs, >> >>>> and >> >>>>> Remote Desktop display to my desktop computer)... >> >>>>> >> >>>>> -- >> >>>>> Robin D. Wilson >> >>>>> Sr. Director of Web Development >> >>>>> KingsIsle Entertainment, Inc. >> >>>>> VOICE: 512-777-1861 >> >>>>> www.KingsIsle.com >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> ------------------------------------------------------------------- >> >>>>> -- To unsubscribe, e-mail: [email protected] >> >>>>> For additional commands, e-mail: [email protected] >> >>>>> >> >>>> >> >>>> >> >>>> --------------------------------------------------------------------- >> >>>> To unsubscribe, e-mail: [email protected] >> >>>> For additional commands, e-mail: [email protected] >> >>>> >> >>>> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: [email protected] >> >> For additional commands, e-mail: [email protected] >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: [email protected] >> >> For additional commands, e-mail: [email protected] >> >> >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] >> > For additional commands, e-mail: [email protected] >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> >
