Le 21/05/2012 22:12, Philippe Mouawad a ecrit : > Hello, > I read recently this little comparison : > https://github.com/excilys/gatling/wiki/Benchmarks<https://github.com/excilys/gatling/wiki/Benchmark> > > I reviewed the test plan that was used to make the test. > It seems to me the test is little biased: > > - View Results Tree is in test plan (as it uses a lot of memory, it's a > big issue) > - View Results in Table (same thing) > - 3 Non Standard JMeter listener, so It's not pure JMeter: > - jp@gc - Transactions per Second > - jp@gc - Response Times Over Time > - jp@gc - Active Threads Over Time > - Default XML output seems to have been used , it's against best > practices > > > Besides the following conclusions that seem to me non scientific and purely > subjective: > > - The testers, new to both Gatling and JMeter found that JMeter was > harder to learn and use than Gatling to create the simulations, despite the > use of a proxy. > > > I will only look at other ideas expressed: > > - JMeter creates one thread per user simulated. If there is not enough > memory allocated to the JVM, it can crash trying to create these threads > - This need to be detailed, cause either it fails with OOM and it's > not during thread creation, either it fails with "unable to create new > native thread" > - For instance, JMeter could not run 1500 users with 512 MB (what was > used for Gatling even with 2000 users); OutOfMemoryErrors are recorded in > the table as *OOM* > - => I made a Test with up to 2000 Threads with 512 m without any > crash, it depends on Test and on application > - Another problem occurred with the 2000 users simulations; it seems > that JMeter can not simulate more than 1514 users independently from the > memory that was allocated to the JVM > - => I made a Test with up to 2000 Threads with 512 m without any > crash, so assertion is false, it depends on Test and on application > > > As it's difficult to install the application used for test (last version > does not seem to work as expected) , if they provide a working WAR against > a local Postgres DB I will be happy to test with it. > But in current state, application seems to be packaged for cloud or H2 > local DB, I didn't want to spend too much time setting up application as I > don't know its real status. > > I just tried to run Test Plan against a blank tomcat to verify what they > say about Thread Creation, I didn't find any issue on this. > > So I decided to make a very simple scenario test on Tomcat Examples (It > goes to Session Example, adds attribute, go back to index, go back to > Session Example, test contains Response assertion for each Request). >
Please, indicate the Tomcat version used? it's the same machine? config of tomcat server? tuning of JVM for Tomcat? OS? OS tuning? JVM editor? JVM editor/version for your JMeter? OS? OS Tuning (TCP tuning?)? Can you post the jmx file? Milamber > It is not at all representative but it is a way for me to check potential > issues in JMeter and performance changes accross versions. > > I ran the test with 1500 VU using JMeter 2.5.1, 2.7 (current trunk) with > -Xmx512m, 10 minutes run and CSV output against a local Tomcat (I restard > tomcat between tests and control its health): > > - I noticed that current trunk version behaves much better in terms of > memory than 2.5.1 or 2.6: > - In 2.5.1 : > - GC activity is much higher with around 5 GC CPU peaks every 2 > minutes, and 20 FULL GC of 700 to 800 ms each > - Throughput: 97,71% > - Pauses : 13,69s > - Mém : 391M/min > - Full GC tend to be much more frequent at end of test > - 2.7: > - no GC CPU peak, 1 FULL GC > - Throughput:98.54 > - Pauses : 8.9s > - 1108m /min > - Summary: > - 25.1: > - 164676 samples in 605,1s > - 272,2/s > - Avg: 97 > - 2.7: > - 165367 in 605.0s > - 273.3/s > - Avg: 228 > - I also noticed results have a much better look: > - in 2.5.1, Transactions/s are around 300 / sec during 4 minutes, > then drop to 200/s, go up to 400/s , then down to 260/s and > finally 200/sec > - in 2.7, Transactions stay around 300/sec > > > > Actions: > > - I think it would be useful to have some reference application on which > we could test JMeter behaviour > - What would the best place to put these comparisons ? wiki ? which > indicators should we put ? > - Further testing should be done against a richer application that could > be deployed locally > > > Regards > Philippe > >
