jvisualvm will give you more in depth cpu profiling capabilities if run from the server. This will give a cpu time per method call. You can also get memory profiling statistics (how much memory is used for each class, how many instances there are of each class, etc.). jvisualvm should be included with your jdk on the server.
The CMS garbage collector can result in pauses in the JVM: http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html "the concurrent collector does most of its tracing and sweeping work with the application threads still running, so only brief pauses are seen by the application threads. However, if the concurrent collector is unable to finish reclaiming the unreachable objects before the tenured generation fills up, or if an allocation cannot be satisfied with the available free space blocks in the tenured generation, then the application is paused and the collection is completed with all the application threads stopped. The inability to complete a collection concurrently is referred to as concurrent mode failure and indicates the need to adjust the concurrent collector parameters." You may want to look into the parallel collector. Look at how quickly each of the generations fills up. If they are filling up too quickly, adjust the storage ratios. If the CMS collector does not have the attributes you desire, look into a different collector. Possibly: XX:+UseParallelGC XX:+UseParallelOldGC XX:ParallelGCThreads= http://stackoverflow.com/questions/574021/experiencing-1-second-pauses-using-useconcmarksweepgc-help Are you using a 64-bit JVM? On Thu, Mar 29, 2012 at 2:48 PM, L G Robinson <[email protected]> wrote: > ** > Hi Axton, > > Thanks for reading and for your suggestions. > > I have done a little bit of JVM monitoring. I used jconsole, since it was > already there and was easy enough to startup. I watched the various heaps > over a weeks time and I didn't see anything alarming. GC seemed to be > happening regularly and doesn't seem to to take very long. These are my > JAVA_OPTS: > > -XX:MaxPermSize=256m > -Xms2048m > -Xmx2048m > -XX:+UseCompressedOops > -XX:+UseConcMarkSweepGC > -XX:+UseParNewGC > -XX:NewRatio=3 > > Apache Tomcat/6.0.32 > JVM: Sun 1.6.0_31-b04 > Linux: 2.6.32-131.17.1.el6.x86_64 > > Your question about how the Mid-tier servers connect to the AR server was > a good one. I believe I am using proxy connections with the default of 80 > as a max. I was using the default queues but I am wondering if there would > be any advantage to setting up a dedicated queue for the Mid-tier servers > to hit. I think I'll give that a try to see if anything changes. Probably > won't, since RRR|Log indicates no thread contention. But maybe those > threads are managed differently. > > Do you agree that using proxy is more efficient or would you expect better > performance if that was disabled? If it is disabled, does the Mid-tier make > a dedicated connection to the AR server for each Mid-tier user? I'm a > little vague on how that works and I was not able to find much in the > documentation about it. Any pointers on where to learn more about that? > > catalina.out shows the following settings: > > - jrpcMode: true > - jniLoadMode: 1 > - apiRecordingMode: 0 > - useConnectionPooling: true > - maxProxiesPerServer: 80 > - idleConnectionsPerServer: 5 > - connectionTimeout: 0 > - connectionLifespan: 0 > - timeUnit: MINUTES > - minimumSupportedServerRpcVersion: 12 > - connectionMaxRetries: 1 > - timeLagBetweenRetriesMillSec: 100 > - stringizeFieldAssignments: false > - stringizeSetIfQualification: false > - useLegacyQualParser: false > - useLegacyAssignParser: false > - useLegacyQualFormatter: false > - useLegacyAssignFormatter: false > - UsePooling: true > - , Pool setting info: , # maxProxyPerServer- 80, # > idleConnectionsPerServer- 5, connectionTimeout setting- 00:00:00, > connectionLifespan setting- 00:00:00 > - Currently, connection pooling true > - Client adjusts connection variables maximumPoolSize = 80 > minIdleProxySize = 5 connectionTimeout = 0 timeUnit = null > - Client sets useConnectionPooling to : true > - Root WebApplicationContext: initialization completed in 2594 ms > - Currently, connection pooling true > - Client sets connection limits per server to :80 > - UsePooling: true > - , Pool setting info: , # maxProxyPerServer- 80, # > idleConnectionsPerServer- 5, connectionTimeout setting- 00:00:00, > connectionLifespan setting- 00:00:00 > - Client sets useConnectionPooling to : true > > > - Connects to ars00srv:33000 through com.bmc.arsys.api.ProxyJRpc@7952506e > - Connects to ars00srv:33000 through com.bmc.arsys.api.ProxyJRpc@60e1e567 > - Connects to ars00srv:33000 through com.bmc.arsys.api.ProxyJRpc@477588d5 > - Connects to ars00srv:33000 through com.bmc.arsys.api.ProxyJRpc@756a7c99 > > Thanks again for your help. > Larry > > > On Thu, Mar 29, 2012 at 11:03 AM, Axton <[email protected]> wrote: > >> ** Well constructed post. It gives me a lot to work with when thinking >> about your problem. I applaud you for that. >> >> The interesting thing is that a restart of either the Tomcat servers or >> the ARServer fixes your issue. This throws an unknown variable into the >> mix (a monkey wrench, of sorts). >> >> If you are inclined to dig into the Tomcat performance, I suggest >> enabling JMX and hooking up a JMX monitoring tool to the Tomcat instances >> (I prefer jvisualvm, which is part of the jdk) to see what the server is >> busy doing and where the memory allocation is going. You might find that >> the CPU time required for garbage collection gets higher and higher. If >> this is the case, depending on the configuration of the Linux VMs (cpu >> count, memory size, etc.) that house Tomcat, you may be able to play with >> the GC settings to alleviate this. It may point you to some issue within >> the midtier that you can't address, but if you know where the problem is >> you can work with support. >> >> If using JMX, you will want to use a local jvisualvm so that you can get >> cpu profiling statistics. >> >> You might also look at how the midtier connects to the arserver. Do you >> use a proxy connection? Do you use a private queue? If you use a private >> queue, how many threads does it have and are those threads congested? >> >> Axton Grams >> >> _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ > _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
