Hello, Thanks for the wiki page.
I've got the test plan, and I run some tests. Target: 1 VM on KVM host | 2 Gb RAM | 4 vCPU Debian 6.0.5 (64 bits) Tomcat 6.0.35 (from Apache site) Sun Java 1.6u32 (64bits) Restart Tomcat before each test Tomcat with default params. JMeter machine (another machine) : 8 Gb RAM | 2 core 2.53GHz Debian 6.0.5 (64 bits) Sun Java 1.6u32 (64bits) JMeter only change on JAVA options: -Xloggc:/tmp/gc_jmeterXXX.txt With the GC log, I've make some stats. With the original test plan: HttpClient 4: JMeter_251 Nb GC: 202 Full GC: 17 Time GC: 13.9376210 Full: 11.3386420 Total: 25.2762630 JMeter_26 Nb GC: 216 Full GC: 18 Time GC: 14.6083630 Full: 12.0604050 Total: 26.6687680 JMeter_27 Nb GC: 199 Full GC: 15 Time GC: 11.7238800 Full: 9.5478390 Total: 21.2717190 Stats results HC4: JMeter 2.5.1: Requests 163917 in 604.8s = 271.0/s Avg: 148 JMeter 2.6: Requests 163208 in 605.0s = 269.7/s Avg: 155 JMeter 2.7RC3: 164035 in 605.1s = 271.1/s Avg: 159 Changes HttpClient to 3.1: JMeter_251 Nb GC: 136 Full GC: 1 Time GC: 8.6780490 Full: .0254120 Total: 8.7034610 JMeter_26 Nb GC: 146 Full GC: 1 Time GC: 9.0567930 Full: .0305490 Total: 9.0873420 JMeter_27 Nb GC: 137 Full GC: 1 Time GC: 6.9212530 Full: .0282870 Total: 6.9495400 Stats results HC3.1: JMeter 2.5.1: Requests 165103 in 605.1s = 272.9/s Avg: 233 JMeter 2.6: Requests 165063 in 605.1s = 272.8/s Avg: 214 JMeter 2.7RC3: Requests 164860 in 604.5s = 272.7/s Avg: 180 Changes to Java http client: JMeter_251 Nb GC: 138 Full GC: 1 Time GC: 8.1128620 Full: .0253190 Total: 8.1381810 JMeter_26 Nb GC: 149 Full GC: 2 Time GC: 8.4811800 Full: .6600440 Total: 9.1412240 JMeter_27 Nb GC: 137 Full GC: 1 Time GC: 6.4912220 Full: .0283680 Total: 6.5195900 Stats results HJ: JMeter 2.5.1: Requests 164966 in 605.0s = 272.7/s Avg: 164 JMeter 2.6: Requests 165069 in 605.2s = 272.8/s Avg: 198 JMeter 2.7RC3: Requests 165233 in 605.0s = 273.1/s Avg: 213 * JMeter 2.7(RC3) and HttpJava requests seems the best with the GC (HC3.1 too) * HttpJava and HC 3.1 seems the best with GC and req/s * HC 4 seems have the best results with the time average * HC 4 seems bring some (big) differences with the java virtual machine. An investigation on HC4 can be done? perhaps send an email on HttpComponents team. Milamber Le 26/05/2012 02:00, Philippe Mouawad a ecrit : > Hello, > I finally managed to insert images and Script: > http://wiki.apache.org/jmeter/JMeterPerformance > > Regards > Philippe > On Fri, May 25, 2012 at 1:48 PM, Philippe Mouawad < > [email protected]> wrote: > > >> Hello, >> And are you able to insert images ? or Test Plan ? >> >> Sebb, I think I tried with Attachment to insert Plan or Image, it proposes >> a link type with a protocol, but I don't see how to upload the image or >> Test Plan. >> >> There is a button Insert/Edit image but it is disabled. >> >> Regards >> Philippe >> >> >> On Fri, May 25, 2012 at 12:06 PM, Milamber <[email protected]> wrote: >> >> >>> On Fri, May 25, 2012 at 10:35 AM, sebb <[email protected]> wrote: >>> >>> >>>> On 25 May 2012 10:31, Milamber <[email protected]> wrote: >>>> >>>>> Hello, >>>>> >>>>> Actuallly, the page >>>>> >>> http://wiki.apache.org/jmeter/JMeterPerformancereturns >>> >>>>> a http 500 error (others pages works) >>>>> >>>> Works for me at present. >>>> >>>> >>> Ok for me too. >>> >>> >>> >>> >>> >>>> >>>>> Milamber >>>>> >>>>> === >>>>> Internal Server Error >>>>> >>>>> The server encountered an internal error or misconfiguration and was >>>>> >>>> unable >>>> >>>>> to complete your request. >>>>> >>>>> Please contact the server administrator at [email protected] >>>>> inform them of the time this error occurred, and the actions you >>>>> >>>> performed >>>> >>>>> just before this error. >>>>> >>>>> More information about this error may be available in the server error >>>>> >>>> log. >>>> >>>>> ------------------------------ >>>>> Apache/2.4.1 (Unix) OpenSSL/1.0.0g Server at wiki.apache.org Port 80 >>>>> >>>>> === >>>>> >>>>> >>>>> >>>>> On Fri, May 25, 2012 at 1:27 AM, sebb <[email protected]> wrote: >>>>> >>>>> >>>>>> On 24 May 2012 22:15, Philippe Mouawad <[email protected]> >>>>>> >>>> wrote: >>>> >>>>>>> Hello, >>>>>>> How can I insert image ? button is disabled in GUI mode. >>>>>>> To insert Test Plan, is it an attachment ? >>>>>>> >>>>>> Try clicking "Attachments" before starting the edit. >>>>>> >>>>>> >>>>>>> Thanks >>>>>>> Regards >>>>>>> Philippe >>>>>>> >>>>>>> On Wed, May 23, 2012 at 12:22 AM, sebb <[email protected]> wrote: >>>>>>> >>>>>>> >>>>>>>> On 22 May 2012 20:18, Philippe Mouawad < >>>>>>>> >>> [email protected]> >>> >>>>>> wrote: >>>>>> >>>>>>>>> Hello Milamber, >>>>>>>>> My responses below. >>>>>>>>> Regards >>>>>>>>> Philippe >>>>>>>>> >>>>>>>>> On Mon, May 21, 2012 at 11:51 PM, Milamber <[email protected] >>>>>>>>> >>>> >>>>>> wrote: >>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Le 21/05/2012 22:12, Philippe Mouawad a ecrit : >>>>>>>>>> >>>>>>>>>>> Hello, >>>>>>>>>>> I read recently this little comparison : >>>>>>>>>>> https://github.com/excilys/gatling/wiki/Benchmarks< >>>>>>>>>>> >>>>>>>>>> https://github.com/excilys/gatling/wiki/Benchmark> >>>>>>>>>> >>>>>>>>>>> I reviewed the test plan that was used to make the test. >>>>>>>>>>> It seems to me the test is little biased: >>>>>>>>>>> >>>>>>>>>>> - View Results Tree is in test plan (as it uses a lot of >>>>>>>>>>> >>>> memory, >>>> >>>>>>>> it's >>>>>>>> >>>>>>>>>> a >>>>>>>>>> >>>>>>>>>>> big issue) >>>>>>>>>>> - View Results in Table (same thing) >>>>>>>>>>> - 3 Non Standard JMeter listener, so It's not pure JMeter: >>>>>>>>>>> - jp@gc - Transactions per Second >>>>>>>>>>> - jp@gc - Response Times Over Time >>>>>>>>>>> - jp@gc - Active Threads Over Time >>>>>>>>>>> - Default XML output seems to have been used , it's >>>>>>>>>>> >>> against >>> >>>> best >>>> >>>>>>>>>>> practices >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Besides the following conclusions that seem to me non >>>>>>>>>>> >>> scientific >>> >>>>>> and >>>>>> >>>>>>>>>> purely >>>>>>>>>> >>>>>>>>>>> subjective: >>>>>>>>>>> >>>>>>>>>>> - The testers, new to both Gatling and JMeter found that >>>>>>>>>>> >>>> JMeter >>>> >>>>>> was >>>>>> >>>>>>>>>>> harder to learn and use than Gatling to create the >>>>>>>>>>> >>>> simulations, >>>> >>>>>>>>>> despite the >>>>>>>>>> >>>>>>>>>>> use of a proxy. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I will only look at other ideas expressed: >>>>>>>>>>> >>>>>>>>>>> - JMeter creates one thread per user simulated. If there >>>>>>>>>>> >>> is >>> >>>> not >>>> >>>>>>>> enough >>>>>>>> >>>>>>>>>>> memory allocated to the JVM, it can crash trying to create >>>>>>>>>>> >>>> these >>>> >>>>>>>>>> threads >>>>>>>>>> >>>>>>>>>>> - This need to be detailed, cause either it fails with >>>>>>>>>>> >>> OOM >>> >>>>>> and >>>>>> >>>>>>>> it's >>>>>>>> >>>>>>>>>>> not during thread creation, either it fails with >>>>>>>>>>> >>> "unable >>> >>>> to >>>> >>>>>>>> create >>>>>>>> >>>>>>>>>> new >>>>>>>>>> >>>>>>>>>>> native thread" >>>>>>>>>>> - For instance, JMeter could not run 1500 users with 512 >>>>>>>>>>> >>> MB >>> >>>>>> (what >>>>>> >>>>>>>> was >>>>>>>> >>>>>>>>>>> used for Gatling even with 2000 users); OutOfMemoryErrors >>>>>>>>>>> >>> are >>> >>>>>>>>>> recorded in >>>>>>>>>> >>>>>>>>>>> the table as *OOM* >>>>>>>>>>> - => I made a Test with up to 2000 Threads with 512 m >>>>>>>>>>> >>>> without >>>> >>>>>>>> any >>>>>>>> >>>>>>>>>>> crash, it depends on Test and on application >>>>>>>>>>> - Another problem occurred with the 2000 users >>>>>>>>>>> >>> simulations; >>> >>>> it >>>> >>>>>>>> seems >>>>>>>> >>>>>>>>>>> that JMeter can not simulate more than 1514 users >>>>>>>>>>> >>>> independently >>>> >>>>>>>> from >>>>>>>> >>>>>>>>>> the >>>>>>>>>> >>>>>>>>>>> memory that was allocated to the JVM >>>>>>>>>>> - => I made a Test with up to 2000 Threads with 512 m >>>>>>>>>>> >>>> without >>>> >>>>>>>> any >>>>>>>> >>>>>>>>>>> crash, so assertion is false, it depends on Test and on >>>>>>>>>>> >>>>>>>> application >>>>>>>> >>>>>>>>>>> >>>>>>>>>>> As it's difficult to install the application used for test >>>>>>>>>>> >>> (last >>> >>>>>>>> version >>>>>>>> >>>>>>>>>>> does not seem to work as expected) , if they provide a >>>>>>>>>>> >>> working >>> >>>> WAR >>>> >>>>>>>>>> against >>>>>>>>>> >>>>>>>>>>> a local Postgres DB I will be happy to test with it. >>>>>>>>>>> But in current state, application seems to be packaged for >>>>>>>>>>> >>> cloud >>> >>>>>> or H2 >>>>>> >>>>>>>>>>> local DB, I didn't want to spend too much time setting up >>>>>>>>>>> >>>>>> application >>>>>> >>>>>>>> as >>>>>>>> >>>>>>>>>> I >>>>>>>>>> >>>>>>>>>>> don't know its real status. >>>>>>>>>>> >>>>>>>>>>> I just tried to run Test Plan against a blank tomcat to >>>>>>>>>>> >>> verify >>> >>>> what >>>> >>>>>>>> they >>>>>>>> >>>>>>>>>>> say about Thread Creation, I didn't find any issue on this. >>>>>>>>>>> >>>>>>>>>>> So I decided to make a very simple scenario test on Tomcat >>>>>>>>>>> >>>> Examples >>>> >>>>>>>> (It >>>>>>>> >>>>>>>>>>> goes to Session Example, adds attribute, go back to index, go >>>>>>>>>>> >>>> back >>>> >>>>>> to >>>>>> >>>>>>>>>>> Session Example, test contains Response assertion for each >>>>>>>>>>> >>>>>> Request). >>>>>> >>>>>>>>>>> >>>>>>>>>> Please, indicate the Tomcat version used? it's the same >>>>>>>>>> >>> machine? >>> >>>>>> config >>>>>> >>>>>>>>>> of tomcat server? tuning of JVM for Tomcat? OS? OS tuning? JVM >>>>>>>>>> >>>>>> editor? >>>>>> >>>>>>>>>> Tomcat apache-tomcat-6.0.24 >>>>>>>>>> >>>>>>>>> -Xms256m -Xmx1024m >>>>>>>>> <Connector port="8080" protocol="HTTP/1.1" >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>> >>> compressableMimeType="text/html,text/xml,text/plain,text/javascript,application/json" >>> >>>>>>>>> compression="off" >>>>>>>>> socketBuffer="8" >>>>>>>>> maxThreads="400" >>>>>>>>> connectionTimeout="20000" >>>>>>>>> redirectPort="8443" /> >>>>>>>>> Set session timeout in web.xml to 1min >>>>>>>>> java version "1.6.0_29" >>>>>>>>> Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-10M3527) >>>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed >>>>>>>>> >>> mode) >>> >>>>>>>>> Mac OS 10.6.8 >>>>>>>>> JMeter and Tomcat on same machine >>>>>>>>> No particular OS Tuning >>>>>>>>> 8GO RAM >>>>>>>>> No swap, nothing runnning except these 2 >>>>>>>>> Tomcat CPU around 5% >>>>>>>>> Memory around 50 mo >>>>>>>>> >>>>>>>>> JVM editor/version for your JMeter? OS? OS Tuning (TCP tuning?)? >>>>>>>>> >>>>>>>>>> Same config >>>>>>>>>> >>>>>>>>> No TCP tuning >>>>>>>>> >>>>>>>>> >>>>>>>>>> Can you post the jmx file? >>>>>>>>>> >>>>>>>>>> Will it be accepted on this list ? >>>>>>>>>> >>>>>>>> Best to create a Wiki page and attach the test file there. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>>> Milamber >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> It is not at all representative but it is a way for me to >>>>>>>>>>> >>> check >>> >>>>>>>> potential >>>>>>>> >>>>>>>>>>> issues in JMeter and performance changes accross versions. >>>>>>>>>>> >>>>>>>>>>> I ran the test with 1500 VU using JMeter 2.5.1, 2.7 (current >>>>>>>>>>> >>>>>> trunk) >>>>>> >>>>>>>> with >>>>>>>> >>>>>>>>>>> -Xmx512m, 10 minutes run and CSV output against a local >>>>>>>>>>> >>> Tomcat >>> >>>> (I >>>> >>>>>>>> restard >>>>>>>> >>>>>>>>>>> tomcat between tests and control its health): >>>>>>>>>>> >>>>>>>>>>> - I noticed that current trunk version behaves much >>>>>>>>>>> >>> better in >>> >>>>>>>> terms of >>>>>>>> >>>>>>>>>>> memory than 2.5.1 or 2.6: >>>>>>>>>>> - In 2.5.1 : >>>>>>>>>>> - GC activity is much higher with around 5 GC CPU peaks >>>>>>>>>>> >>>>>> every 2 >>>>>> >>>>>>>>>>> minutes, and 20 FULL GC of 700 to 800 ms each >>>>>>>>>>> - Throughput: 97,71% >>>>>>>>>>> - Pauses : 13,69s >>>>>>>>>>> - Mém : 391M/min >>>>>>>>>>> - Full GC tend to be much more frequent at end of >>>>>>>>>>> >>> test >>> >>>>>>>>>>> - 2.7: >>>>>>>>>>> - no GC CPU peak, 1 FULL GC >>>>>>>>>>> - Throughput:98.54 >>>>>>>>>>> - Pauses : 8.9s >>>>>>>>>>> - 1108m /min >>>>>>>>>>> - Summary: >>>>>>>>>>> - 25.1: >>>>>>>>>>> - 164676 samples in 605,1s >>>>>>>>>>> - 272,2/s >>>>>>>>>>> - Avg: 97 >>>>>>>>>>> - 2.7: >>>>>>>>>>> - 165367 in 605.0s >>>>>>>>>>> - 273.3/s >>>>>>>>>>> - Avg: 228 >>>>>>>>>>> - I also noticed results have a much better look: >>>>>>>>>>> - in 2.5.1, Transactions/s are around 300 / sec during >>>>>>>>>>> >>> 4 >>> >>>>>>>> minutes, >>>>>>>> >>>>>>>>>>> then drop to 200/s, go up to 400/s , then down to 260/s >>>>>>>>>>> >>>> and >>>> >>>>>>>>>>> finally 200/sec >>>>>>>>>>> - in 2.7, Transactions stay around 300/sec >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Actions: >>>>>>>>>>> >>>>>>>>>>> - I think it would be useful to have some reference >>>>>>>>>>> >>>> application >>>> >>>>>> on >>>>>> >>>>>>>>>> which >>>>>>>>>> >>>>>>>>>>> we could test JMeter behaviour >>>>>>>>>>> - What would the best place to put these comparisons ? >>>>>>>>>>> >>> wiki ? >>> >>>>>> which >>>>>> >>>>>>>>>>> indicators should we put ? >>>>>>>>>>> - Further testing should be done against a richer >>>>>>>>>>> >>> application >>> >>>>>> that >>>>>> >>>>>>>>>> could >>>>>>>>>> >>>>>>>>>>> be deployed locally >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards >>>>>>>>>>> Philippe >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Cordialement. >>>>>>>>> Philippe Mouawad. >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Cordialement. >>>>>>> Philippe Mouawad. >>>>>>> >>>>>> >>>> >>> >> >> >> -- >> Cordialement. >> Philippe Mouawad. >> >> >> >> >> > >
