On Sun, 2012-05-27 at 16:47 +0100, Milamber wrote: > > > Le 27/05/2012 14:57, Oleg Kalnichevski a ecrit : > > On Sat, 2012-05-26 at 23:14 +0100, Milamber wrote: > > > > > Le 26/05/2012 21:25, Milamber a ecrit : > > > > > > > Hello, > > > > > > > > Thanks for the wiki page. > > > > > > > > I've got the test plan, and I run some tests. > > > > > > > > Target: > > > > 1 VM on KVM host | 2 Gb RAM | 4 vCPU > > > > Debian 6.0.5 (64 bits) > > > > Tomcat 6.0.35 (from Apache site) > > > > Sun Java 1.6u32 (64bits) > > > > Restart Tomcat before each test > > > > Tomcat with default params. > > > > > > > > JMeter machine (another machine) : > > > > 8 Gb RAM | 2 core 2.53GHz > > > > Debian 6.0.5 (64 bits) > > > > Sun Java 1.6u32 (64bits) > > > > JMeter only change on JAVA options: -Xloggc:/tmp/gc_jmeterXXX.txt > > > > > > > > With the GC log, I've make some stats. > > > > > > > > With the original test plan: HttpClient 4: > > > > (==>4.1) > > > > JMeter_251 Nb GC: 202 Full GC: 17 Time GC: 13.9376210 Full: > > > > 11.3386420 Total: 25.2762630 > > > > JMeter_26 Nb GC: 216 Full GC: 18 Time GC: 14.6083630 Full: > > > > 12.0604050 Total: 26.6687680 > > > > JMeter_27 Nb GC: 199 Full GC: 15 Time GC: 11.7238800 Full: > > > > 9.5478390 Total: 21.2717190 > > > > > > > > Stats results HC4: (==>4.1) > > > > JMeter 2.5.1: Requests 163917 in 604.8s = 271.0/s Avg: 148 > > > > JMeter 2.6: Requests 163208 in 605.0s = 269.7/s Avg: 155 > > > > JMeter 2.7RC3: 164035 in 605.1s = 271.1/s Avg: 159 > > > > > > > > > > > With JMeter trunk and HttpClient 4.2 (core, client, mime): > > > > > > JMeter_trunk_hc42 Nb GC: 131 Full GC: 1 Time GC: 5.7903490 > > > Full: .0285440 Total: 5.8188930 > > > > > > JMeter trunk with HC 4.2 (not commit): Nb Requests 164762 in 605.2s = > > > 272.3/s Avg: 250 > > > > > > > > > Results (GC counts and avg req/s) with the HC 4.2 are same with HC3.1 > > > and Java. > > > > > > => HC 4.2 seems fixed a performance issue with JVM memory? > > > > > > => Release 2.7.1 to do with HC4.2 ? (or update HC and make a RC4?) > > > > > > Milamber > > > > > > > > > > > > > Changes HttpClient to 3.1: > > > > JMeter_251 Nb GC: 136 Full GC: 1 Time GC: 8.6780490 Full: > > > > .0254120 Total: 8.7034610 > > > > JMeter_26 Nb GC: 146 Full GC: 1 Time GC: 9.0567930 Full: > > > > .0305490 Total: 9.0873420 > > > > JMeter_27 Nb GC: 137 Full GC: 1 Time GC: 6.9212530 Full: > > > > .0282870 Total: 6.9495400 > > > > > > > > Stats results HC3.1: > > > > JMeter 2.5.1: Requests 165103 in 605.1s = 272.9/s Avg: 233 > > > > JMeter 2.6: Requests 165063 in 605.1s = 272.8/s Avg: 214 > > > > JMeter 2.7RC3: Requests 164860 in 604.5s = 272.7/s Avg: 180 > > > > > > > > Changes to Java http client: > > > > JMeter_251 Nb GC: 138 Full GC: 1 Time GC: 8.1128620 Full: > > > > .0253190 Total: 8.1381810 > > > > JMeter_26 Nb GC: 149 Full GC: 2 Time GC: 8.4811800 Full: > > > > .6600440 Total: 9.1412240 > > > > JMeter_27 Nb GC: 137 Full GC: 1 Time GC: 6.4912220 Full: > > > > .0283680 Total: 6.5195900 > > > > > > > > Stats results HJ: > > > > JMeter 2.5.1: Requests 164966 in 605.0s = 272.7/s Avg: 164 > > > > JMeter 2.6: Requests 165069 in 605.2s = 272.8/s Avg: 198 > > > > JMeter 2.7RC3: Requests 165233 in 605.0s = 273.1/s Avg: 213 > > > > > > > > > > > > * JMeter 2.7(RC3) and HttpJava requests seems the best with the GC > > > > (HC3.1 too) > > > > * HttpJava and HC 3.1 seems the best with GC and req/s > > > > * HC 4 seems have the best results with the time average > > > > * HC 4 seems bring some (big) differences with the java virtual machine. > > > > > > > > An investigation on HC4 can be done? perhaps send an email on > > > > HttpComponents team. > > > > > > > > Milamber > > > > > > > > > > > > > > I cannot think of any changes in the 4.2 branch that could make a big > > difference in terms of performance or memory footprint compared to 4.1. > > It also seems somewhat strange that the 4.1 based sampler appears to be > > generating massively more intermediate garbage while still managing to > > perform at the same level as other samplers (unless I am misreading the > > results somehow). > > > > Do you know what kind of objects cause excessive garbage collection when > > running the 4.1 sampler? > > > > Hello Oleg, > > I've make some heap dump during tests (with oracle jvisualvm) > I uses Eclipse Memory Analyzer to analyze the heap. > I've found perhaps a > "org.apache.http.impl.conn.SingleClientConnManager" issue. > > With JMeter 2.7RC3 (HttpClient 4.1) : > > Biggest Top-Level Dominator Classes > > Label > Number of > Objects > Used Heap Size > Retained Heap > Size > Retained Heap, > % > org.apache.jmeter.protocol.http.control.HeaderManager > 6,000 > 384,000 > 40,142,160 > 18.26% > org.apache.http.impl.conn.SingleClientConnManager > 1,500 > 120,000 > 33,289,328 > 15.14% > org.apache.jmeter.threads.JMeterThread > 1,500 > 216,000 > 31,718,400 > 14.43% > org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy > 6,004 > 384,256 > 30,097,176 > 13.69% > java.lang.Thread > 1,510 > 265,760 > 18,165,056 > 8.26% > org.apache.jmeter.assertions.ResponseAssertion > 6,000 > 336,000 > 9,504,000 > 4.32% > org.apache.jmeter.threads.JMeterContext > 1,500 > 120,000 > 8,230,528 > 3.74% > org.apache.jmeter.sampler.TestAction > 6,000 > 336,000 > 7,680,000 > 3.49% > org.apache.jmeter.threads.ThreadGroup > 1,501 > 96,064 > 5,379,528 > 2.45% > org.apache.jmeter.config.ConfigTestElement > 1,500 > 84,000 > 4,648,080 > 2.11% > org.apache.jmeter.protocol.http.control.CookieManager > 1,501 > 108,072 > 4,296,984 > 1.95% > java.lang.Class > 1,759 > 27,840 > 4,177,528 > 1.90% > org.apache.http.conn.scheme.SchemeRegistry > 1,500 > 36,000 > 3,725,520 > 1.69% > org.apache.jmeter.config.Arguments > 1,500 > 84,000 > 3,180,000 > 1.45% > org.apache.jmeter.control.TransactionController > 1,500 > 204,000 > 2,832,000 > 1.29% > > * Total: > 15 > entries > 40,775 > 2,801,992 > 207,066,288 > > > > > > With JMeter Trunk + HttpClient 4.2 : > > > Biggest Top-Level Dominator Classes > Label > Number of > Objects > Used Heap Size > Retained Heap > Size > Retained Heap, > % > org.apache.jmeter.protocol.http.control.HeaderManager > 6,000 > 384,000 > 41,283,936 > 22.75% > org.apache.jmeter.threads.JMeterThread > 1,500 > 216,000 > 31,925,472 > 17.59% > org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy > 6,004 > 384,256 > 28,151,656 > 15.51% > java.lang.Thread > 1,504 > 264,704 > 15,899,168 > 8.76% > org.apache.jmeter.assertions.ResponseAssertion > 6,000 > 336,000 > 9,441,672 > 5.20% > org.apache.jmeter.sampler.TestAction > 6,000 > 336,000 > 7,601,096 > 4.19% > org.apache.jmeter.threads.JMeterContext > 1,332 > 106,560 > 7,156,624 > 3.94% > org.apache.jmeter.threads.ThreadGroup > 1,501 > 96,064 > 5,353,896 > 2.95% > org.apache.jmeter.config.ConfigTestElement > 1,500 > 84,000 > 5,141,184 > 2.83% > org.apache.jmeter.protocol.http.control.CookieManager > 1,501 > 108,072 > 4,582,536 > 2.52% > org.apache.http.impl.conn.BasicClientConnectionManager > 1,332 > 85,248 > 4,256,488 > 2.35% > java.lang.Class > 1,228 > 22,200 > 4,006,160 > 2.21% > org.apache.http.conn.scheme.SchemeRegistry > 1,332 > 31,968 > 3,314,016 > 1.83% > org.apache.jmeter.config.Arguments > 1,500 > 84,000 > 3,161,184 > 1.74% > org.apache.jmeter.control.TransactionController > 1,500 > 204,000 > 2,737,920 > 1.51% > org.apache.jmeter.control.LoopController > 1,500 > 144,000 > 2,150,592 > 1.18% > > * Total: > 16 > entries > 41,234 > 2,887,072 > 176,163,600 > > > > > > > When I observed in the HC4.1/JMeter, with outgoing reference for > SingleClientConnManager : > connManager org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl > > Class Name > | Shallow Heap | Retained Heap > ------------------------------------------------------------------------------------------------------------------------------------------ > class org.apache.http.impl.conn.SingleClientConnManager @ 0xd8bd20a0 > | 8 | 336 > |- <class> org.apache.http.impl.conn.SingleClientConnManager @ > 0xe6be9450 | 80 | > 27,008 > | |- connManager org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl > $4 @ 0xe6be9320 | 168 | 9,536 > | | |- value java.util.HashMap$Entry @ 0xe6be9258 > | 48 | 9,784 > | | | '- [10] java.util.HashMap$Entry[16] @ 0xe6be9208 > | 152 | 9,936 > | | | '- table java.util.HashMap @ 0xe6be91d8 > | 64 | 10,000 > | | | '- value java.lang.ThreadLocal$ThreadLocalMap$Entry @ > 0xe6be91b8 | 56 | 10,056 > | | | '- [4] java.lang.ThreadLocal$ThreadLocalMap > $Entry[16] @ 0xe3fed728 | 152 | > 11,064 > | | | '- table java.lang.ThreadLocal$ThreadLocalMap @ > 0xe3fed710 | 32 | 11,096 > | | | '- threadLocals java.lang.Thread @ 0xe3fe4b00 > Thread Group 1-1483 Thread | 176 | 11,592 > | | | |- [1486] java.lang.Thread[2048] @ > 0xe3218f40 | 16,408 | > 16,408 > | | | | '- threads java.lang.ThreadGroup @ > 0xe0000030 main | 72 | > 16,552 > | | | |- <Java Local>, me java.lang.Thread @ > 0xe3fe4b00 Thread Group 1-1483 Thread | 176 | > 11,592 > | | | | |- [1486] java.lang.Thread[2048] @ > 0xe3218f40 | 16,408 | > 16,408 > | | | | |- <Java Local>, me java.lang.Thread @ > 0xe3fe4b00 Thread Group 1-1483 Thread| 176 | 11,592 > | | | | |- value > java.util.concurrent.ConcurrentHashMap$HashEntry @ 0xe4417ba0 | > 48 | 48 > | | | | '- Total: 3 entries > | | > | | | |- value > java.util.concurrent.ConcurrentHashMap$HashEntry @ 0xe4417ba0 > | 48 | 48 > | | | '- Total: 3 entries > | | > | | |- this$1 org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl$4 > $1 @ 0xe6bee448 | 32 | 32 > | | '- Total: 2 entries > | | > | |- referent java.lang.ref.Finalizer @ 0xe6bf2eb0 > | 64 | 113,152 > | |- this$0 org.apache.http.impl.conn.SingleClientConnManager > $PoolEntry @ 0xe844eee0 | 64 | > 26,896 > | '- Total: 3 entries > | | > |- <class> org.apache.http.impl.conn.SingleClientConnManager @ > 0xe6be6d88 | 80 | > 27,008 > |- <class> org.apache.http.impl.conn.SingleClientConnManager @ > 0xe6be40c8 | 80 | > 27,008 > |- <class> org.apache.http.impl.conn.SingleClientConnManager @ > 0xe6be19c8 | 80 | > 80 > |- <class> org.apache.http.impl.conn.SingleClientConnManager @ > 0xe6bde940 | 80 | > 27,008 > |- <class> org.apache.http.impl.conn.SingleClientConnManager @ > 0xe6bdc268 | 80 | > 26,984 > ------------------------------------------------------------------------------------------------------------------------------------------ > Class Name | Shallow Heap | Retained Heap > --------------------------------------------------------- > Total: 25 of 1,501 entries| | > --------------------------------------------------------- > > Not observed with JMeter trunk with HC4.2. > > Perhaps, the new implementation [1] [2] of ConnectionManager with HC42 > is better ? > >
The new connection managers are definitely better in terms of code quality. However, I did not expect them to be any faster, though, as they pretty much shared the same design with the old ones. At the same time it does appear that BasicClientConnectionManager tends to have a much smaller footprint judging by the heap dumps, which is a pleasant surprise. What is not quite clear to me is why JMeter creates so many connection managers at all. Does it create a separate connection manager per virtual user or per request? In any case this explains why HttpJava tends to be faster in your tests. HttpJava uses one global connection pool for all HttpUrlConnection instances. Therefore it uses fewer physical connections to execute the same number of requests which usually is always faster. This would be equivalent to HttpClient using one pooling connection manager for all virtual users and requests. Hope this helps interpret the numbers. Oleg > The TestPlan from the wiki: > http://wiki.apache.org/jmeter/JMeterPerformance?action=AttachFile&do=view&target=TestPlan.jmx > > Run from JMeter 2.7RC3: > http://people.apache.org/~milamber/jmeter-2.7RC3/dist/ > > Run from JMeter trung with HC4.2 (temporary) > http://people.apache.org/~milamber/temp/ > > Milamber > > > > [1] > http://www.apache.org/dist/httpcomponents/httpcore/RELEASE_NOTES.txt > [2] > http://www.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES.txt > > > > > > > Oleg > > > > > > > > > > Le 26/05/2012 02:00, Philippe Mouawad a ecrit : > > > > > > > > > > > > > Hello, > > > > > I finally managed to insert images and Script: > > > > > http://wiki.apache.org/jmeter/JMeterPerformance > > > > > > > > > > Regards > > > > > Philippe > > > > > On Fri, May 25, 2012 at 1:48 PM, Philippe Mouawad < > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > And are you able to insert images ? or Test Plan ? > > > > > > > > > > > > Sebb, I think I tried with Attachment to insert Plan or Image, it > > > > > > proposes > > > > > > a link type with a protocol, but I don't see how to upload the > > > > > > image or > > > > > > Test Plan. > > > > > > > > > > > > There is a button Insert/Edit image but it is disabled. > > > > > > > > > > > > Regards > > > > > > Philippe > > > > > > > > > > > > > > > > > > On Fri, May 25, 2012 at 12:06 PM, Milamber <[email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 25, 2012 at 10:35 AM, sebb <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 25 May 2012 10:31, Milamber <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > Actuallly, the page > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://wiki.apache.org/jmeter/JMeterPerformancereturns > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > a http 500 error (others pages works) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Works for me at present. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ok for me too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Milamber > > > > > > > > > > > > > > > > > > === > > > > > > > > > Internal Server Error > > > > > > > > > > > > > > > > > > The server encountered an internal error or misconfiguration > > > > > > > > > and was > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > unable > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to complete your request. > > > > > > > > > > > > > > > > > > Please contact the server administrator at > > > > > > > > > [email protected] > > > > > > > > > inform them of the time this error occurred, and the actions > > > > > > > > > you > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > performed > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > just before this error. > > > > > > > > > > > > > > > > > > More information about this error may be available in the > > > > > > > > > server error > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > log. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------ > > > > > > > > > Apache/2.4.1 (Unix) OpenSSL/1.0.0g Server at wiki.apache.org > > > > > > > > > Port 80 > > > > > > > > > > > > > > > > > > === > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 25, 2012 at 1:27 AM, sebb <[email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 24 May 2012 22:15, Philippe Mouawad > > > > > > > > > > <[email protected]> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > How can I insert image ? button is disabled in GUI mode. > > > > > > > > > > > To insert Test Plan, is it an attachment ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Try clicking "Attachments" before starting the edit. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > Regards > > > > > > > > > > > Philippe > > > > > > > > > > > > > > > > > > > > > > On Wed, May 23, 2012 at 12:22 AM, sebb <[email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 22 May 2012 20:18, Philippe Mouawad < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hello Milamber, > > > > > > > > > > > > > My responses below. > > > > > > > > > > > > > Regards > > > > > > > > > > > > > Philippe > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 21, 2012 at 11:51 PM, Milamber > > > > > > > > > > > > > <[email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Le 21/05/2012 22:12, Philippe Mouawad a ecrit : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > I read recently this little comparison : > > > > > > > > > > > > > > > https://github.com/excilys/gatling/wiki/Benchmarks< > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/excilys/gatling/wiki/Benchmark> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I reviewed the test plan that was used to make > > > > > > > > > > > > > > > the test. > > > > > > > > > > > > > > > It seems to me the test is little biased: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - View Results Tree is in test plan (as it > > > > > > > > > > > > > > > uses a lot of > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > memory, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > it's > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > big issue) > > > > > > > > > > > > > > > - View Results in Table (same thing) > > > > > > > > > > > > > > > - 3 Non Standard JMeter listener, so It's not > > > > > > > > > > > > > > > pure JMeter: > > > > > > > > > > > > > > > - jp@gc - Transactions per Second > > > > > > > > > > > > > > > - jp@gc - Response Times Over Time > > > > > > > > > > > > > > > - jp@gc - Active Threads Over Time > > > > > > > > > > > > > > > - Default XML output seems to have been used , > > > > > > > > > > > > > > > it's > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > against > > > > > > > > > > > > > > > > > > > > > > > > > > > > > best > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > practices > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Besides the following conclusions that seem to me > > > > > > > > > > > > > > > non > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > scientific > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > purely > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > subjective: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - The testers, new to both Gatling and JMeter > > > > > > > > > > > > > > > found that > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > JMeter > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > was > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > harder to learn and use than Gatling to create the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > simulations, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > despite the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > use of a proxy. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I will only look at other ideas expressed: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - JMeter creates one thread per user > > > > > > > > > > > > > > > simulated. If there > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > enough > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > memory allocated to the JVM, it can crash trying > > > > > > > > > > > > > > > to create > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > these > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > threads > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - This need to be detailed, cause either it fails > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > OOM > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > it's > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > not during thread creation, either it fails with > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "unable > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > create > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > native thread" > > > > > > > > > > > > > > > - For instance, JMeter could not run 1500 > > > > > > > > > > > > > > > users with 512 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > MB > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (what > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > was > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > used for Gatling even with 2000 users); > > > > > > > > > > > > > > > OutOfMemoryErrors > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > recorded in > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the table as *OOM* > > > > > > > > > > > > > > > - => I made a Test with up to 2000 Threads > > > > > > > > > > > > > > > with 512 m > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > without > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > any > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > crash, it depends on Test and on application > > > > > > > > > > > > > > > - Another problem occurred with the 2000 users > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > simulations; > > > > > > > > > > > > > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > seems > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > that JMeter can not simulate more than 1514 users > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > independently > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > memory that was allocated to the JVM > > > > > > > > > > > > > > > - => I made a Test with up to 2000 Threads > > > > > > > > > > > > > > > with 512 m > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > without > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > any > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > crash, so assertion is false, it depends on Test > > > > > > > > > > > > > > > and on > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > application > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As it's difficult to install the application used > > > > > > > > > > > > > > > for test > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (last > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > version > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > does not seem to work as expected) , if they > > > > > > > > > > > > > > > provide a > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > working > > > > > > > > > > > > > > > > > > > > > > > > > > > > > WAR > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > against > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > a local Postgres DB I will be happy to test with > > > > > > > > > > > > > > > it. > > > > > > > > > > > > > > > But in current state, application seems to be > > > > > > > > > > > > > > > packaged for > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cloud > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > or H2 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > local DB, I didn't want to spend too much time > > > > > > > > > > > > > > > setting up > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > application > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > don't know its real status. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I just tried to run Test Plan against a blank > > > > > > > > > > > > > > > tomcat to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > verify > > > > > > > > > > > > > > > > > > > > > > > > > > > > > what > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > they > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > say about Thread Creation, I didn't find any > > > > > > > > > > > > > > > issue on this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > So I decided to make a very simple scenario test > > > > > > > > > > > > > > > on Tomcat > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Examples > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (It > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > goes to Session Example, adds attribute, go back > > > > > > > > > > > > > > > to index, go > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > back > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Session Example, test contains Response assertion > > > > > > > > > > > > > > > for each > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Request). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please, indicate the Tomcat version used? it's the > > > > > > > > > > > > > > same > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > machine? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > config > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > of tomcat server? tuning of JVM for Tomcat? OS? OS > > > > > > > > > > > > > > tuning? JVM > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > editor? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Tomcat apache-tomcat-6.0.24 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Xms256m -Xmx1024m > > > > > > > > > > > > > <Connector port="8080" protocol="HTTP/1.1" > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > compressableMimeType="text/html,text/xml,text/plain,text/javascript,application/json" > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > compression="off" > > > > > > > > > > > > > socketBuffer="8" > > > > > > > > > > > > > maxThreads="400" > > > > > > > > > > > > > connectionTimeout="20000" > > > > > > > > > > > > > redirectPort="8443" /> > > > > > > > > > > > > > Set session timeout in web.xml to 1min > > > > > > > > > > > > > java version "1.6.0_29" > > > > > > > > > > > > > Java(TM) SE Runtime Environment (build > > > > > > > > > > > > > 1.6.0_29-b11-402-10M3527) > > > > > > > > > > > > > Java HotSpot(TM) 64-Bit Server VM (build > > > > > > > > > > > > > 20.4-b02-402, mixed > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > mode) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Mac OS 10.6.8 > > > > > > > > > > > > > JMeter and Tomcat on same machine > > > > > > > > > > > > > No particular OS Tuning > > > > > > > > > > > > > 8GO RAM > > > > > > > > > > > > > No swap, nothing runnning except these 2 > > > > > > > > > > > > > Tomcat CPU around 5% > > > > > > > > > > > > > Memory around 50 mo > > > > > > > > > > > > > > > > > > > > > > > > > > JVM editor/version for your JMeter? OS? OS Tuning > > > > > > > > > > > > > (TCP tuning?)? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Same config > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > No TCP tuning > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Can you post the jmx file? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Will it be accepted on this list ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best to create a Wiki page and attach the test file > > > > > > > > > > > > there. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Milamber > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It is not at all representative but it is a way > > > > > > > > > > > > > > > for me to > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > check > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > potential > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > issues in JMeter and performance changes accross > > > > > > > > > > > > > > > versions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I ran the test with 1500 VU using JMeter 2.5.1, > > > > > > > > > > > > > > > 2.7 (current > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > trunk) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Xmx512m, 10 minutes run and CSV output against a > > > > > > > > > > > > > > > local > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Tomcat > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (I > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > restard > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > tomcat between tests and control its health): > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - I noticed that current trunk version behaves > > > > > > > > > > > > > > > much > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > better in > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > terms of > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > memory than 2.5.1 or 2.6: > > > > > > > > > > > > > > > - In 2.5.1 : > > > > > > > > > > > > > > > - GC activity is much higher with around 5 > > > > > > > > > > > > > > > GC CPU peaks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > every 2 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > minutes, and 20 FULL GC of 700 to 800 ms each > > > > > > > > > > > > > > > - Throughput: 97,71% > > > > > > > > > > > > > > > - Pauses : 13,69s > > > > > > > > > > > > > > > - Mém : 391M/min > > > > > > > > > > > > > > > - Full GC tend to be much more frequent > > > > > > > > > > > > > > > at end of > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > test > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - 2.7: > > > > > > > > > > > > > > > - no GC CPU peak, 1 FULL GC > > > > > > > > > > > > > > > - Throughput:98.54 > > > > > > > > > > > > > > > - Pauses : 8.9s > > > > > > > > > > > > > > > - 1108m /min > > > > > > > > > > > > > > > - Summary: > > > > > > > > > > > > > > > - 25.1: > > > > > > > > > > > > > > > - 164676 samples in 605,1s > > > > > > > > > > > > > > > - 272,2/s > > > > > > > > > > > > > > > - Avg: 97 > > > > > > > > > > > > > > > - 2.7: > > > > > > > > > > > > > > > - 165367 in 605.0s > > > > > > > > > > > > > > > - 273.3/s > > > > > > > > > > > > > > > - Avg: 228 > > > > > > > > > > > > > > > - I also noticed results have a much > > > > > > > > > > > > > > > better look: > > > > > > > > > > > > > > > - in 2.5.1, Transactions/s are around 300 / > > > > > > > > > > > > > > > sec during > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 4 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > minutes, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > then drop to 200/s, go up to 400/s , then down to > > > > > > > > > > > > > > > 260/s > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > finally 200/sec > > > > > > > > > > > > > > > - in 2.7, Transactions stay around 300/sec > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Actions: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - I think it would be useful to have some > > > > > > > > > > > > > > > reference > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > application > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > we could test JMeter behaviour > > > > > > > > > > > > > > > - What would the best place to put these > > > > > > > > > > > > > > > comparisons ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wiki ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > indicators should we put ? > > > > > > > > > > > > > > > - Further testing should be done against a > > > > > > > > > > > > > > > richer > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > application > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > could > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > be deployed locally > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > Philippe > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > Cordialement. > > > > > > > > > > > > > Philippe Mouawad. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Cordialement. > > > > > > > > > > > Philippe Mouawad. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Cordialement. > > > > > > Philippe Mouawad. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
