Le 26/05/2012 21:25, Milamber a ecrit : > Hello, > > Thanks for the wiki page. > > I've got the test plan, and I run some tests. > > Target: > 1 VM on KVM host | 2 Gb RAM | 4 vCPU > Debian 6.0.5 (64 bits) > Tomcat 6.0.35 (from Apache site) > Sun Java 1.6u32 (64bits) > Restart Tomcat before each test > Tomcat with default params. > > JMeter machine (another machine) : > 8 Gb RAM | 2 core 2.53GHz > Debian 6.0.5 (64 bits) > Sun Java 1.6u32 (64bits) > JMeter only change on JAVA options: -Xloggc:/tmp/gc_jmeterXXX.txt > > With the GC log, I've make some stats. > > With the original test plan: HttpClient 4: > (==>4.1) > JMeter_251 Nb GC: 202 Full GC: 17 Time GC: 13.9376210 Full: > 11.3386420 Total: 25.2762630 > JMeter_26 Nb GC: 216 Full GC: 18 Time GC: 14.6083630 Full: > 12.0604050 Total: 26.6687680 > JMeter_27 Nb GC: 199 Full GC: 15 Time GC: 11.7238800 Full: > 9.5478390 Total: 21.2717190 > > Stats results HC4: (==>4.1) > JMeter 2.5.1: Requests 163917 in 604.8s = 271.0/s Avg: 148 > JMeter 2.6: Requests 163208 in 605.0s = 269.7/s Avg: 155 > JMeter 2.7RC3: 164035 in 605.1s = 271.1/s Avg: 159 >
With JMeter trunk and HttpClient 4.2 (core, client, mime): JMeter_trunk_hc42 Nb GC: 131 Full GC: 1 Time GC: 5.7903490 Full: .0285440 Total: 5.8188930 JMeter trunk with HC 4.2 (not commit): Nb Requests 164762 in 605.2s = 272.3/s Avg: 250 Results (GC counts and avg req/s) with the HC 4.2 are same with HC3.1 and Java. => HC 4.2 seems fixed a performance issue with JVM memory? => Release 2.7.1 to do with HC4.2 ? (or update HC and make a RC4?) Milamber > Changes HttpClient to 3.1: > JMeter_251 Nb GC: 136 Full GC: 1 Time GC: 8.6780490 Full: > .0254120 Total: 8.7034610 > JMeter_26 Nb GC: 146 Full GC: 1 Time GC: 9.0567930 Full: > .0305490 Total: 9.0873420 > JMeter_27 Nb GC: 137 Full GC: 1 Time GC: 6.9212530 Full: > .0282870 Total: 6.9495400 > > Stats results HC3.1: > JMeter 2.5.1: Requests 165103 in 605.1s = 272.9/s Avg: 233 > JMeter 2.6: Requests 165063 in 605.1s = 272.8/s Avg: 214 > JMeter 2.7RC3: Requests 164860 in 604.5s = 272.7/s Avg: 180 > > Changes to Java http client: > JMeter_251 Nb GC: 138 Full GC: 1 Time GC: 8.1128620 Full: > .0253190 Total: 8.1381810 > JMeter_26 Nb GC: 149 Full GC: 2 Time GC: 8.4811800 Full: > .6600440 Total: 9.1412240 > JMeter_27 Nb GC: 137 Full GC: 1 Time GC: 6.4912220 Full: > .0283680 Total: 6.5195900 > > Stats results HJ: > JMeter 2.5.1: Requests 164966 in 605.0s = 272.7/s Avg: 164 > JMeter 2.6: Requests 165069 in 605.2s = 272.8/s Avg: 198 > JMeter 2.7RC3: Requests 165233 in 605.0s = 273.1/s Avg: 213 > > > * JMeter 2.7(RC3) and HttpJava requests seems the best with the GC > (HC3.1 too) > * HttpJava and HC 3.1 seems the best with GC and req/s > * HC 4 seems have the best results with the time average > * HC 4 seems bring some (big) differences with the java virtual machine. > > An investigation on HC4 can be done? perhaps send an email on > HttpComponents team. > > Milamber > > > Le 26/05/2012 02:00, Philippe Mouawad a ecrit : > >> Hello, >> I finally managed to insert images and Script: >> http://wiki.apache.org/jmeter/JMeterPerformance >> >> Regards >> Philippe >> On Fri, May 25, 2012 at 1:48 PM, Philippe Mouawad < >> [email protected]> wrote: >> >> >> >>> Hello, >>> And are you able to insert images ? or Test Plan ? >>> >>> Sebb, I think I tried with Attachment to insert Plan or Image, it proposes >>> a link type with a protocol, but I don't see how to upload the image or >>> Test Plan. >>> >>> There is a button Insert/Edit image but it is disabled. >>> >>> Regards >>> Philippe >>> >>> >>> On Fri, May 25, 2012 at 12:06 PM, Milamber <[email protected]> wrote: >>> >>> >>> >>>> On Fri, May 25, 2012 at 10:35 AM, sebb <[email protected]> wrote: >>>> >>>> >>>> >>>>> On 25 May 2012 10:31, Milamber <[email protected]> wrote: >>>>> >>>>> >>>>>> Hello, >>>>>> >>>>>> Actuallly, the page >>>>>> >>>>>> >>>> http://wiki.apache.org/jmeter/JMeterPerformancereturns >>>> >>>> >>>>>> a http 500 error (others pages works) >>>>>> >>>>>> >>>>> Works for me at present. >>>>> >>>>> >>>>> >>>> Ok for me too. >>>> >>>> >>>> >>>> >>>> >>>> >>>>> >>>>> >>>>>> Milamber >>>>>> >>>>>> === >>>>>> Internal Server Error >>>>>> >>>>>> The server encountered an internal error or misconfiguration and was >>>>>> >>>>>> >>>>> unable >>>>> >>>>> >>>>>> to complete your request. >>>>>> >>>>>> Please contact the server administrator at [email protected] >>>>>> inform them of the time this error occurred, and the actions you >>>>>> >>>>>> >>>>> performed >>>>> >>>>> >>>>>> just before this error. >>>>>> >>>>>> More information about this error may be available in the server error >>>>>> >>>>>> >>>>> log. >>>>> >>>>> >>>>>> ------------------------------ >>>>>> Apache/2.4.1 (Unix) OpenSSL/1.0.0g Server at wiki.apache.org Port 80 >>>>>> >>>>>> === >>>>>> >>>>>> >>>>>> >>>>>> On Fri, May 25, 2012 at 1:27 AM, sebb <[email protected]> wrote: >>>>>> >>>>>> >>>>>> >>>>>>> On 24 May 2012 22:15, Philippe Mouawad <[email protected]> >>>>>>> >>>>>>> >>>>> wrote: >>>>> >>>>> >>>>>>>> Hello, >>>>>>>> How can I insert image ? button is disabled in GUI mode. >>>>>>>> To insert Test Plan, is it an attachment ? >>>>>>>> >>>>>>>> >>>>>>> Try clicking "Attachments" before starting the edit. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Thanks >>>>>>>> Regards >>>>>>>> Philippe >>>>>>>> >>>>>>>> On Wed, May 23, 2012 at 12:22 AM, sebb <[email protected]> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On 22 May 2012 20:18, Philippe Mouawad < >>>>>>>>> >>>>>>>>> >>>> [email protected]> >>>> >>>> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>>>>> Hello Milamber, >>>>>>>>>> My responses below. >>>>>>>>>> Regards >>>>>>>>>> Philippe >>>>>>>>>> >>>>>>>>>> On Mon, May 21, 2012 at 11:51 PM, Milamber <[email protected] >>>>>>>>>> >>>>>>>>>> >>>>> >>>>> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Le 21/05/2012 22:12, Philippe Mouawad a ecrit : >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Hello, >>>>>>>>>>>> I read recently this little comparison : >>>>>>>>>>>> https://github.com/excilys/gatling/wiki/Benchmarks< >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> https://github.com/excilys/gatling/wiki/Benchmark> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> I reviewed the test plan that was used to make the test. >>>>>>>>>>>> It seems to me the test is little biased: >>>>>>>>>>>> >>>>>>>>>>>> - View Results Tree is in test plan (as it uses a lot of >>>>>>>>>>>> >>>>>>>>>>>> >>>>> memory, >>>>> >>>>> >>>>>>>>> it's >>>>>>>>> >>>>>>>>> >>>>>>>>>>> a >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> big issue) >>>>>>>>>>>> - View Results in Table (same thing) >>>>>>>>>>>> - 3 Non Standard JMeter listener, so It's not pure JMeter: >>>>>>>>>>>> - jp@gc - Transactions per Second >>>>>>>>>>>> - jp@gc - Response Times Over Time >>>>>>>>>>>> - jp@gc - Active Threads Over Time >>>>>>>>>>>> - Default XML output seems to have been used , it's >>>>>>>>>>>> >>>>>>>>>>>> >>>> against >>>> >>>> >>>>> best >>>>> >>>>> >>>>>>>>>>>> practices >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Besides the following conclusions that seem to me non >>>>>>>>>>>> >>>>>>>>>>>> >>>> scientific >>>> >>>> >>>>>>> and >>>>>>> >>>>>>> >>>>>>>>>>> purely >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> subjective: >>>>>>>>>>>> >>>>>>>>>>>> - The testers, new to both Gatling and JMeter found that >>>>>>>>>>>> >>>>>>>>>>>> >>>>> JMeter >>>>> >>>>> >>>>>>> was >>>>>>> >>>>>>> >>>>>>>>>>>> harder to learn and use than Gatling to create the >>>>>>>>>>>> >>>>>>>>>>>> >>>>> simulations, >>>>> >>>>> >>>>>>>>>>> despite the >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> use of a proxy. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I will only look at other ideas expressed: >>>>>>>>>>>> >>>>>>>>>>>> - JMeter creates one thread per user simulated. If there >>>>>>>>>>>> >>>>>>>>>>>> >>>> is >>>> >>>> >>>>> not >>>>> >>>>> >>>>>>>>> enough >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> memory allocated to the JVM, it can crash trying to create >>>>>>>>>>>> >>>>>>>>>>>> >>>>> these >>>>> >>>>> >>>>>>>>>>> threads >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> - This need to be detailed, cause either it fails with >>>>>>>>>>>> >>>>>>>>>>>> >>>> OOM >>>> >>>> >>>>>>> and >>>>>>> >>>>>>> >>>>>>>>> it's >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> not during thread creation, either it fails with >>>>>>>>>>>> >>>>>>>>>>>> >>>> "unable >>>> >>>> >>>>> to >>>>> >>>>> >>>>>>>>> create >>>>>>>>> >>>>>>>>> >>>>>>>>>>> new >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> native thread" >>>>>>>>>>>> - For instance, JMeter could not run 1500 users with 512 >>>>>>>>>>>> >>>>>>>>>>>> >>>> MB >>>> >>>> >>>>>>> (what >>>>>>> >>>>>>> >>>>>>>>> was >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> used for Gatling even with 2000 users); OutOfMemoryErrors >>>>>>>>>>>> >>>>>>>>>>>> >>>> are >>>> >>>> >>>>>>>>>>> recorded in >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> the table as *OOM* >>>>>>>>>>>> - => I made a Test with up to 2000 Threads with 512 m >>>>>>>>>>>> >>>>>>>>>>>> >>>>> without >>>>> >>>>> >>>>>>>>> any >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> crash, it depends on Test and on application >>>>>>>>>>>> - Another problem occurred with the 2000 users >>>>>>>>>>>> >>>>>>>>>>>> >>>> simulations; >>>> >>>> >>>>> it >>>>> >>>>> >>>>>>>>> seems >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> that JMeter can not simulate more than 1514 users >>>>>>>>>>>> >>>>>>>>>>>> >>>>> independently >>>>> >>>>> >>>>>>>>> from >>>>>>>>> >>>>>>>>> >>>>>>>>>>> the >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> memory that was allocated to the JVM >>>>>>>>>>>> - => I made a Test with up to 2000 Threads with 512 m >>>>>>>>>>>> >>>>>>>>>>>> >>>>> without >>>>> >>>>> >>>>>>>>> any >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> crash, so assertion is false, it depends on Test and on >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> application >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> As it's difficult to install the application used for test >>>>>>>>>>>> >>>>>>>>>>>> >>>> (last >>>> >>>> >>>>>>>>> version >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> does not seem to work as expected) , if they provide a >>>>>>>>>>>> >>>>>>>>>>>> >>>> working >>>> >>>> >>>>> WAR >>>>> >>>>> >>>>>>>>>>> against >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> a local Postgres DB I will be happy to test with it. >>>>>>>>>>>> But in current state, application seems to be packaged for >>>>>>>>>>>> >>>>>>>>>>>> >>>> cloud >>>> >>>> >>>>>>> or H2 >>>>>>> >>>>>>> >>>>>>>>>>>> local DB, I didn't want to spend too much time setting up >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>> application >>>>>>> >>>>>>> >>>>>>>>> as >>>>>>>>> >>>>>>>>> >>>>>>>>>>> I >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> don't know its real status. >>>>>>>>>>>> >>>>>>>>>>>> I just tried to run Test Plan against a blank tomcat to >>>>>>>>>>>> >>>>>>>>>>>> >>>> verify >>>> >>>> >>>>> what >>>>> >>>>> >>>>>>>>> they >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> say about Thread Creation, I didn't find any issue on this. >>>>>>>>>>>> >>>>>>>>>>>> So I decided to make a very simple scenario test on Tomcat >>>>>>>>>>>> >>>>>>>>>>>> >>>>> Examples >>>>> >>>>> >>>>>>>>> (It >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> goes to Session Example, adds attribute, go back to index, go >>>>>>>>>>>> >>>>>>>>>>>> >>>>> back >>>>> >>>>> >>>>>>> to >>>>>>> >>>>>>> >>>>>>>>>>>> Session Example, test contains Response assertion for each >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>> Request). >>>>>>> >>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> Please, indicate the Tomcat version used? it's the same >>>>>>>>>>> >>>>>>>>>>> >>>> machine? >>>> >>>> >>>>>>> config >>>>>>> >>>>>>> >>>>>>>>>>> of tomcat server? tuning of JVM for Tomcat? OS? OS tuning? JVM >>>>>>>>>>> >>>>>>>>>>> >>>>>>> editor? >>>>>>> >>>>>>> >>>>>>>>>>> Tomcat apache-tomcat-6.0.24 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> -Xms256m -Xmx1024m >>>>>>>>>> <Connector port="8080" protocol="HTTP/1.1" >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>>> compressableMimeType="text/html,text/xml,text/plain,text/javascript,application/json" >>>> >>>> >>>>>>>>>> compression="off" >>>>>>>>>> socketBuffer="8" >>>>>>>>>> maxThreads="400" >>>>>>>>>> connectionTimeout="20000" >>>>>>>>>> redirectPort="8443" /> >>>>>>>>>> Set session timeout in web.xml to 1min >>>>>>>>>> java version "1.6.0_29" >>>>>>>>>> Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-10M3527) >>>>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed >>>>>>>>>> >>>>>>>>>> >>>> mode) >>>> >>>> >>>>>>>>>> Mac OS 10.6.8 >>>>>>>>>> JMeter and Tomcat on same machine >>>>>>>>>> No particular OS Tuning >>>>>>>>>> 8GO RAM >>>>>>>>>> No swap, nothing runnning except these 2 >>>>>>>>>> Tomcat CPU around 5% >>>>>>>>>> Memory around 50 mo >>>>>>>>>> >>>>>>>>>> JVM editor/version for your JMeter? OS? OS Tuning (TCP tuning?)? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Same config >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> No TCP tuning >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Can you post the jmx file? >>>>>>>>>>> >>>>>>>>>>> Will it be accepted on this list ? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> Best to create a Wiki page and attach the test file there. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Milamber >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> It is not at all representative but it is a way for me to >>>>>>>>>>>> >>>>>>>>>>>> >>>> check >>>> >>>> >>>>>>>>> potential >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> issues in JMeter and performance changes accross versions. >>>>>>>>>>>> >>>>>>>>>>>> I ran the test with 1500 VU using JMeter 2.5.1, 2.7 (current >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>> trunk) >>>>>>> >>>>>>> >>>>>>>>> with >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> -Xmx512m, 10 minutes run and CSV output against a local >>>>>>>>>>>> >>>>>>>>>>>> >>>> Tomcat >>>> >>>> >>>>> (I >>>>> >>>>> >>>>>>>>> restard >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> tomcat between tests and control its health): >>>>>>>>>>>> >>>>>>>>>>>> - I noticed that current trunk version behaves much >>>>>>>>>>>> >>>>>>>>>>>> >>>> better in >>>> >>>> >>>>>>>>> terms of >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> memory than 2.5.1 or 2.6: >>>>>>>>>>>> - In 2.5.1 : >>>>>>>>>>>> - GC activity is much higher with around 5 GC CPU peaks >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>> every 2 >>>>>>> >>>>>>> >>>>>>>>>>>> minutes, and 20 FULL GC of 700 to 800 ms each >>>>>>>>>>>> - Throughput: 97,71% >>>>>>>>>>>> - Pauses : 13,69s >>>>>>>>>>>> - Mém : 391M/min >>>>>>>>>>>> - Full GC tend to be much more frequent at end of >>>>>>>>>>>> >>>>>>>>>>>> >>>> test >>>> >>>> >>>>>>>>>>>> - 2.7: >>>>>>>>>>>> - no GC CPU peak, 1 FULL GC >>>>>>>>>>>> - Throughput:98.54 >>>>>>>>>>>> - Pauses : 8.9s >>>>>>>>>>>> - 1108m /min >>>>>>>>>>>> - Summary: >>>>>>>>>>>> - 25.1: >>>>>>>>>>>> - 164676 samples in 605,1s >>>>>>>>>>>> - 272,2/s >>>>>>>>>>>> - Avg: 97 >>>>>>>>>>>> - 2.7: >>>>>>>>>>>> - 165367 in 605.0s >>>>>>>>>>>> - 273.3/s >>>>>>>>>>>> - Avg: 228 >>>>>>>>>>>> - I also noticed results have a much better look: >>>>>>>>>>>> - in 2.5.1, Transactions/s are around 300 / sec during >>>>>>>>>>>> >>>>>>>>>>>> >>>> 4 >>>> >>>> >>>>>>>>> minutes, >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> then drop to 200/s, go up to 400/s , then down to 260/s >>>>>>>>>>>> >>>>>>>>>>>> >>>>> and >>>>> >>>>> >>>>>>>>>>>> finally 200/sec >>>>>>>>>>>> - in 2.7, Transactions stay around 300/sec >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Actions: >>>>>>>>>>>> >>>>>>>>>>>> - I think it would be useful to have some reference >>>>>>>>>>>> >>>>>>>>>>>> >>>>> application >>>>> >>>>> >>>>>>> on >>>>>>> >>>>>>> >>>>>>>>>>> which >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> we could test JMeter behaviour >>>>>>>>>>>> - What would the best place to put these comparisons ? >>>>>>>>>>>> >>>>>>>>>>>> >>>> wiki ? >>>> >>>> >>>>>>> which >>>>>>> >>>>>>> >>>>>>>>>>>> indicators should we put ? >>>>>>>>>>>> - Further testing should be done against a richer >>>>>>>>>>>> >>>>>>>>>>>> >>>> application >>>> >>>> >>>>>>> that >>>>>>> >>>>>>> >>>>>>>>>>> could >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> be deployed locally >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Regards >>>>>>>>>>>> Philippe >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Cordialement. >>>>>>>>>> Philippe Mouawad. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Cordialement. >>>>>>>> Philippe Mouawad. >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>>> >>>> >>> >>> -- >>> Cordialement. >>> Philippe Mouawad. >>> >>> >>> >>> >>> >>> >> >> > >
