Hello,
I just wanted to updated everyone not involved with the Cosmo - Chandler
testing yesterday to the progress made.
Now installed on cosmo-demo.osafoundation.org port 8090 is Cosmo 0.2.4
with the memory leak patch running on BEA's jrockit Java VM.
*** The Cosmo port 8090 install should be used from now on to test
against the Chandler .6 release ****
At the start of testing I had the server configure similarly to one
running with the Sun Java VM on cosmo-demo.osafoundation.org:443 which
allocates 800 megabytes of RAM to the heap:
cosmo-demo.osafoundation.org:443 configuration:
=====================================
/usr/local/java-1.5/bin/java -Xms800m -Xmx800m -server
-Dcom.sun.management.jmxremote -Dical4j.unfolding.relaxed=true
-Dderby.stream.error.file=logs/derby.log -Dderby.infolog.append=true
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.endorsed.dirs=/home/cosmo-demo-roots/latest/tomcat/common/endorsed
-classpath
:/home/cosmo-demo-roots/latest/tomcat/bin/bootstrap.jar:/home/cosmo-demo-roots/latest/tomcat/bin/commons-logging-api.jar
-Dcatalina.base=/home/cosmo-demo-roots/latest/tomcat
-Dcatalina.home=/home/cosmo-demo-roots/latest/tomcat
-Djava.io.tmpdir=/home/cosmo-demo-roots/latest/tomcat/temp
org.apache.catalina.startup.Bootstrap start
initial cosmo-demo.osafoundation.org:8090 configuration:
=========================================
/home/bkirsch/jrockit/jre/bin/java -xgcprio:pausetime
-Xpausetarget=400ms -Xns:650m -Xms:800m -Xmx:800m -Xgcpause -Xgcreport
-Xverbose:memory -Xverboselog:/home/bkirsch/logs/gc.log -Xmanagement
-server -Dcom.sun.management.jmxremote -Dical4j.unfolding.relaxed=true
-Dderby.stream.error.file=logs/derby.log -Dderby.infolog.append=true
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.endorsed.dirs=/home/bkirsch/cosmo-0.2.4/tomcat/common/endorsed
-classpath
:/home/bkirsch/cosmo-0.2.4/tomcat/bin/bootstrap.jar:/home/bkirsch/cosmo-0.2.4/tomcat/bin/commons-logging-api.jar
-Dcatalina.base=/home/bkirsch/cosmo-0.2.4/tomcat
-Dcatalina.home=/home/bkirsch/cosmo-0.2.4/tomcat
-Djava.io.tmpdir=/home/bkirsch/cosmo-0.2.4/tomcat/temp
org.apache.catalina.startup.Bootstrap start
What we saw during the first phase of testing on port 8090 was very slow
response times and the server hanging after around 5 minutes of use.
Looking at the gc.log showed that with 800mgs of RAM allocated to the
heap the server was quickly using the 800mgs and then taking between one
to two minutes to garbage collect a portion of that memory. During the
garbage collection no other java threads can run so the perceived result
was that the server had hung even though once the garbage collection was
complete Cosmo was ready to process more requests. In addition, the
jrockit gc output indicated that considerable page faults were occurring
greatly reducing performance.
So we stopped testing and I rebooted the port 8090 server with the
following changes:
/home/bkirsch/jrockit/jre/bin/java -Xms:200m -Xmx:200m -Xgcpause
-Xgcreport -Xverbose:memory -Xverboselog:/home/bkirsch/logs/gc.log
-Xmanagement -server -Dcom.sun.management.jmxremote
-Dical4j.unfolding.relaxed=true -Dderby.stream.error.file=logs/derby.log
-Dderby.infolog.append=true
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.endorsed.dirs=/home/bkirsch/cosmo-0.2.4/tomcat/common/endorsed
-classpath
:/home/bkirsch/cosmo-0.2.4/tomcat/bin/bootstrap.jar:/home/bkirsch/cosmo-0.2.4/tomcat/bin/commons-logging-api.jar
-Dcatalina.base=/home/bkirsch/cosmo-0.2.4/tomcat
-Dcatalina.home=/home/bkirsch/cosmo-0.2.4/tomcat
-Djava.io.tmpdir=/home/bkirsch/cosmo-0.2.4/tomcat/temp
org.apache.catalina.startup.Bootstrap start
Although the number of garbage collections increased dramatically each
garbage collection took roughly 22 milliseconds instead of one large
garbage collection which paused the app for one to two minutes. In
addition no more page faults occurred.
I removed the -xgcprio:pausetime -Xpausetarget=400ms as well. This flags
told jvm to try and limit gc collection times to no more than 400
milliseconds. This flag was not working as the gc was taking one to two
minutes.
After the changes were made, the responsiveness of Cosmo improved by 75
to a 100 percent and for the rest of the testing session no crashes or
timeout errors were encountered.
This is very encouraging news!
Although I set the VM to use a 200 megabyte Java heap, I did this to
test what a dramatic decrease in the heap size would do to performance.
And as we found out it increased it dramatically.
I have since fined tuned the VM settings and the port 8090 server is
running with the following configuration:
====================================
/home/bkirsch/jrockit/jre/bin/java -xgcprio:throughput -Xms:400m
-Xmx:400m -Xgcpause -Xgcreport -Xverbose:memory
-Xverboselog:/home/bkirsch/logs/gc.log -Xmanagement -server
-Dcom.sun.management.jmxremote -Dical4j.unfolding.relaxed=true
-Dderby.stream.error.file=logs/derby.log -Dderby.infolog.append=true
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.endorsed.dirs=/home/bkirsch/cosmo-0.2.4/tomcat/common/endorsed
-classpath
:/home/bkirsch/cosmo-0.2.4/tomcat/bin/bootstrap.jar:/home/bkirsch/cosmo-0.2.4/tomcat/bin/commons-logging-api.jar
-Dcatalina.base=/home/bkirsch/cosmo-0.2.4/tomcat
-Dcatalina.home=/home/bkirsch/cosmo-0.2.4/tomcat
-Djava.io.tmpdir=/home/bkirsch/cosmo-0.2.4/tomcat/temp
org.apache.catalina.startup.Bootstrap start
It would be great if we could host one more Chandler - Cosmo testing
session before the release to further fine tune performance.
-Brian
--
Brian Kirsch - Email Framework Engineer
Open Source Applications Foundation
543 Howard St. 5th Floor
San Francisco, CA 94105
(415) 946-3056
http://www.osafoundation.org
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/dev
