RE: Issues | Suggestion any?
Howdy, #You're putting too many things in one message. ;) Can't help that...;), in fact I would like to add some more things here You could help it by starting multiple threads of questions instead of one. I bet you'd get more responses ;) Yeah right but there is a catch here. Because we are using the dynamic scheme the connections can go beyond the 100 connections which is the The dynamic scheme is evil ;) Try fixed-wait. Why I thought that it was waiting for closing the logical connections was because the thread dump had some pointers and this time I am enclosing those snapshots. Thread-360 prio=5 tid=0x00ac4f50 nid=0xc99 waiting for monitor entry [b108..b1081998] at oracle.jdbc.pool.OracleConnectionEventListener.connectionClosed(OracleC o nnectionEventListener.java:125) - waiting to lock 0xd4d2c820 (a I see what you mean. Looks like a problem with the Oracle pool. Since I avoid the oracle pooling implementation if possible, I can't help you much there. Not an easy fix, there are many java classes and this will take a lot of time. We want to keep this thing as a long term plan under application improvement heading. Well, it's your application and your plan ;) But to keep a memory leak fix under long term planning is a fairly atrocious management choice IMHO. No, this will have negative impact on the application because in the past its was observed under such conditions of low memory, the GC activity stops (increased pause time) the application and there is no OK. So you know it's thrashing if you only give it 256MB, and yet don't want it to take 1GB. Have you tried values in the middle, e.g. 512MB max heap? rule. Problem is that there is high thread count when there is authorised content is downloaded. These downloads are of the size of 1-5 MB. Has these issues been solved in tomcat 4.0 or higher? I will check it otherwise. (Hell lot of things to do!!!) High thread count is not a bad thing necessarily. You can control the number of request processing threads tomcat spawns via the Connector configuration in server.xml. There are no known issues with large downloads in tomcat 4.x. 2 - 12 tomcat having only one shared code base which is mounted on an NFS (this is too much, one might say). I just want to know if all the error logs (Error logger thread exact) are also written to the same NFS part this will call for high I/O contention (right???) because the This is bad for performance. Install tomcat locally on the server, keep the logs locally. You can have another program periodically transfer the logs to a central location. I am really impressed that you have given me some response...and expect more...haven't heard much from anybody else from the group. That's because you're asking for too much at once. People tend to not response to long complicated messages with many questions. I am personally interested in this situation because I've had large-scale application problems with oracle connection pooling and garbage collection. Last but not least needed to know, if we can have some support contract from apache group to support our application. Is that possible? Not from the ASF directly, no. But there are companies and individuals out there who'd help you for pay. Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Issues | Suggestion any?
servers each. This had infact caused the total memory free on Hostname1 to as low as 182Mb at one occasion where the total memory of the machine was 4GB. I am attaching a file that can help give you some thing to comment on - GC Logs. Thats all for now... ;-) Thanks in advance ~AC -Original Message- From: Shapira, Yoav [mailto:[EMAIL PROTECTED] Sent: Friday, October 24, 2003 12:55 AM To: Tomcat Users List Subject: RE: Issues | Suggestion any? Howdy, I wish your mail program would quote properly -- it's much tougher to read your message without knowing what you wrote and what you quoted ;( Yes we are doing a -Xmx, -Xms, -XX:NewSize and not as I typed, sorry about the confusion. We are in the process to use either of the parallel GC algorithm with jvm 1.4.2 but dont have grounds to prove it would be better but only theoretical (wish if you can point to some). But, we need a parallel collector as we have 4 CPU's per machine and that in fact would help it with some more parameters like compaction. You would never be able to find GC numbers specific to your app. You must benchmark it yourself, on your servers, to get accurate numbers. Only that will tell you which GC settings are best. At the scale you're dealing with, the characteristics of your application matter a whole lot to the GC performance. I don't have at present the idea about this number (778m). But, I was more You should know why you're limited the heap at that unusual number. interested to understand the output that pmap returns and am enclosing one with this mail right now: I'm not a pmap expert, I've never found it useful in debugging performance or GC problems. I'm sure other people on the list could help more in this area. The output marked in red, what does this actually signify ??? Excess swapping ??? Why is the heap size in pmap output not equal to the one under the column size in 'TOP' ??? Top lists a lot more than just the heap, so you can't compare the output from top to just the heap in another tool's output. There was nothing marked in red in your message, so I don't know what you mean by the red question. Anyways what I meant was, if swapping was causing some problems here and do we require more memory or do we need to tune the application more. Comments??? When you look at top, it tells you how much swap space is being used. You want to minimize the amount of swap space used. If you have more physical memory, let the JVM use it by increasing the -Xmx setting. You have your causality mixed up here. High GC does not cause a high number of threads. Yeah I think you are right that high threads don't cause the GC but the default % heap which when filled will invoke the GC. But, more or less all these threads account for database connections actually and to some downloads. That's not what I said: What I said and what you say above are both right, but not the same thing. Why are they unable to come back? Are they not released back into the pool? Do they go bad but not abandoned by the pool? This is what I wanted to know but seems like the connections are held No one can answer that except you. Run your app inside a profiler and see what holds references to the connection objects. The crash message and id was checked. This was found an active bug on the sun's bug database. But, seems they have corrected it in jdk1.4.2. As we have't had any kills all of a sudden yet, it may have been solved or ...?? ... or you've been lucky so far. You should have the latest OS patches for the JDK installed as a matter of proper procedure. Keep watching this, but if it's OK for now, focus more on the memory usage. Please help me find a way out of this or a checklist of what need to be done/checked at this stage. A checklist ;) ;) Amusing. I used to go out with a girl who worked at Sapient. Are you up here in Boston? Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Issues | Suggestion any?
Howdy, You're putting too many things in one message. ;) Oracle Connection Pool We have observed that the number of connections during site outages are going beyond 1000 connections. These 1000 connections are jointly held by 12 tomcats with each tomcat having a separate pool. The min amd max connections are set to 30 and 100 respectively. The connections are not So with your configuration, you'd always have between 30*12 = 360 and 100*12 = 1200 open database connections. Is this really what you want? If your DB can't handle 1000 connections, much less 1200, configure your pools so that they don't reach this max. dumps of the respective tomcats JVM, it was observed that there were a lot of thread waiting on monitor. This we suspect is nothing but a lock all these threads are trying to acquire for closing the logical connection and have the threaddumps pointing to this too (not sure!!!). Not sure is right. Threads could be waiting on a lot of things, and it's not always bad to wait (e.g. idle HTTP processor threads). Which threads were waiting and was it for a common monitor address? Since they are all stuck and in the wait state for longer period, it seems there might be some issue with the oracle connection pool. There is one oracle connection cleanup thread that runs too and seems that this is getting killed. Moreover we have recently upgraded from What is killing the oracle reaper? jdk1.3.1 to jdk1.4.2 and wondering if the jdbc drivers are compatible with the new JVM. The version we are using of oracle is 8.1.7.3 and classes12.zip. I don't know whether this would be right mailing list but wanted to know if there are any known bugs for classes12.zip and if the What is the date of the drivers in classes12.zip? And you've renamed it classes12.jar, right? There are better drivers that come with Oracle 9i (v9.2.0.1), the file is called ojdbc14.jar, and it's backwards-compatible for Oracle 8i. You should try these. Memory leaks in the application are continuosly causing high memory usage of the production servers. One observation which was observed on Then fix the memory leaks. during the week was that amount of memory was reducing in a day to day scenario. This was observed from the 'TOP' display. Moreover, looking at the GC logs of tomcats it was found that memory was on the rise on the The JVM will use as much memory as you let it. Monotonic increase is to be expected when you look at it via the top command. What you're more interested in is the %free of the heap. See how well your apps run if you reduce max memory from 1GB to say 256MB. occasion where the total memory of the machine was 4GB. I am attaching a file that can help give you some thing to comment on - GC Logs. Attachments didn't come through, and a GC log wouldn't be very meaningful to us anyways. Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Issues | Suggestion any?
Hi Shapira y and tomcat users, #You're putting too many things in one message. ;) Can't help that...;), in fact I would like to add some more things here :). #So with your configuration, you'd always have between 30*12 = 360 and #100*12 = 1200 open database connections. Is this really what you want? #If your DB can't handle 1000 connections, much less 1200, configure your #pools so that they don't reach this max. Yeah right but there is a catch here. Because we are using the dynamic scheme the connections can go beyond the 100 connections which is the max connections. Seems, the problem comes up when the number of connections cross 250 on any individual tomcat. What we want is whenever there is this surge of load the connection pool should be able to request the same. I don't have pointers which says that 200-250 connections size of oracle connection pool is not supported by classes12.jar and only have observation. #Not sure is right. Threads could be waiting on a lot of things, and #it's not always bad to wait (e.g. idle HTTP processor threads). Which #threads were waiting and was it for a common monitor address? Why I thought that it was waiting for closing the logical connections was because the thread dump had some pointers and this time I am enclosing those snapshots. Thread-360 prio=5 tid=0x00ac4f50 nid=0xc99 waiting for monitor entry [b108..b1081998] at oracle.jdbc.pool.OracleConnectionEventListener.connectionClosed(OracleCo nnectionEventListener.java:125) - waiting to lock 0xd4d2c820 (a oracle.jdbc.pool.OracleConnectionEventListener) at oracle.jdbc.pool.OraclePooledConnection.callListener(OraclePooledConnect ion.java:482) at oracle.jdbc.pool.OraclePooledConnection.logicalClose(OraclePooledConnect ion.java:445) - locked 0xcc4f5700 (a oracle.jdbc.pool.OraclePooledConnection) at oracle.jdbc.driver.OracleConnection.logicalClose(OracleConnection.java:2 900) - locked 0xcc4f57c0 (a oracle.jdbc.driver.OracleConnection) at oracle.jdbc.driver.OracleConnection.close(OracleConnection.java:1418) - locked 0xcc4f57c0 (a oracle.jdbc.driver.OracleConnection) Thread-360 prio=5 tid=0x00ac4f50 nid=0xc99 waiting for monitor entry [b108..b1081998] at oracle.jdbc.pool.OracleConnectionEventListener.connectionClosed(OracleCo nnectionEventListener.java:125) - waiting to lock 0xd4d2c820 (a oracle.jdbc.pool.OracleConnectionEventListener) at oracle.jdbc.pool.OraclePooledConnection.callListener(OraclePooledConnect ion.java:482) at oracle.jdbc.pool.OraclePooledConnection.logicalClose(OraclePooledConnect ion.java:445) - locked 0xcc4f5700 (a oracle.jdbc.pool.OraclePooledConnection) at oracle.jdbc.driver.OracleConnection.logicalClose(OracleConnection.java:2 900) - locked 0xcc4f57c0 (a oracle.jdbc.driver.OracleConnection) at oracle.jdbc.driver.OracleConnection.close(OracleConnection.java:1418) - locked 0xcc4f57c0 (a oracle.jdbc.driver.OracleConnection) Thread-358 prio=5 tid=0x00ac37a0 nid=0xc97 waiting for monitor entry [b128..b1281998] at oracle.jdbc.pool.OracleConnectionEventListener.connectionClosed(OracleCo nnectionEventListener.java:125) - waiting to lock 0xd4d2c820 (a oracle.jdbc.pool.OracleConnectionEventListener) at oracle.jdbc.pool.OraclePooledConnection.callListener(OraclePooledConnect ion.java:482) at oracle.jdbc.pool.OraclePooledConnection.logicalClose(OraclePooledConnect ion.java:445) - locked 0xc51d5eb8 (a oracle.jdbc.pool.OraclePooledConnection) at oracle.jdbc.driver.OracleConnection.logicalClose(OracleConnection.java:2 900) - locked 0xc51d5f78 (a oracle.jdbc.driver.OracleConnection) at oracle.jdbc.driver.OracleConnection.close(OracleConnection.java:1418) - locked 0xc51d5f78 (a oracle.jdbc.driver.OracleConnection) Thread-357 prio=5 tid=0x00ac30c0 nid=0xc96 runnable [b138..b1381998] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:183) at java.io.BufferedInputStream.read1(BufferedInputStream.java:222) at java.io.BufferedInputStream.read(BufferedInputStream.java:277) - locked 0xc73bdcf0 (a java.io.BufferedInputStream) Thread-356 prio=5 tid=0x00ac2a38 nid=0xc95 waiting for monitor entry [b148..b1481998] at oracle.jdbc.pool.OracleConnectionEventListener.connectionClosed(OracleCo nnectionEventListener.java:125) - waiting to lock 0xd4d2c820 (a oracle.jdbc.pool.OracleConnectionEventListener) at oracle.jdbc.pool.OraclePooledConnection.callListener(OraclePooledConnect ion.java:482) at oracle.jdbc.pool.OraclePooledConnection.logicalClose(OraclePooledConnect ion.java:445) - locked 0xc9ffd1c8 (a oracle.jdbc.pool.OraclePooledConnection) at
RE: Issues | Suggestion any?
Hey, Thanks again, but I was wondering why were you so amused about the Checklist part??? I wanted to know the best practices or some good learned lessons if any for tomcat. Anyways...my situation hasn't improved and all the tomcats have reached the max heap size and they have started taking more time in GC (tenured generation of 10-20 seconds - all the processes running the JVM have touched 1GB in memory). I am not in Boston for sure ;) Girl in sapient you have gone out with ... ;) ... hmmm Arnab -Original Message- From: Shapira, Yoav [mailto:[EMAIL PROTECTED] Sent: Friday, October 24, 2003 12:55 AM To: Tomcat Users List Subject: RE: Issues | Suggestion any? Howdy, I wish your mail program would quote properly -- it's much tougher to read your message without knowing what you wrote and what you quoted ;( Yes we are doing a -Xmx, -Xms, -XX:NewSize and not as I typed, sorry about the confusion. We are in the process to use either of the parallel GC algorithm with jvm 1.4.2 but dont have grounds to prove it would be better but only theoretical (wish if you can point to some). But, we need a parallel collector as we have 4 CPU's per machine and that in fact would help it with some more parameters like compaction. You would never be able to find GC numbers specific to your app. You must benchmark it yourself, on your servers, to get accurate numbers. Only that will tell you which GC settings are best. At the scale you're dealing with, the characteristics of your application matter a whole lot to the GC performance. I don't have at present the idea about this number (778m). But, I was more You should know why you're limited the heap at that unusual number. interested to understand the output that pmap returns and am enclosing one with this mail right now: I'm not a pmap expert, I've never found it useful in debugging performance or GC problems. I'm sure other people on the list could help more in this area. The output marked in red, what does this actually signify ??? Excess swapping ??? Why is the heap size in pmap output not equal to the one under the column size in 'TOP' ??? Top lists a lot more than just the heap, so you can't compare the output from top to just the heap in another tool's output. There was nothing marked in red in your message, so I don't know what you mean by the red question. Anyways what I meant was, if swapping was causing some problems here and do we require more memory or do we need to tune the application more. Comments??? When you look at top, it tells you how much swap space is being used. You want to minimize the amount of swap space used. If you have more physical memory, let the JVM use it by increasing the -Xmx setting. You have your causality mixed up here. High GC does not cause a high number of threads. Yeah I think you are right that high threads don't cause the GC but the default % heap which when filled will invoke the GC. But, more or less all these threads account for database connections actually and to some downloads. That's not what I said: What I said and what you say above are both right, but not the same thing. Why are they unable to come back? Are they not released back into the pool? Do they go bad but not abandoned by the pool? This is what I wanted to know but seems like the connections are held No one can answer that except you. Run your app inside a profiler and see what holds references to the connection objects. The crash message and id was checked. This was found an active bug on the sun's bug database. But, seems they have corrected it in jdk1.4.2. As we have't had any kills all of a sudden yet, it may have been solved or ...?? ... or you've been lucky so far. You should have the latest OS patches for the JDK installed as a matter of proper procedure. Keep watching this, but if it's OK for now, focus more on the memory usage. Please help me find a way out of this or a checklist of what need to be done/checked at this stage. A checklist ;) ;) Amusing. I used to go out with a girl who worked at Sapient. Are you up here in Boston? Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Issues | Suggestion any?
Howdy, The (4) production servers are running Solaris OS with 4CPU, 4GB RAM and 7GB of swap space. In all we have 12 tomcats and 4 Apaches. Each machine is equipped with one apache and 3 tomcats. The Database machines is also running Solaris with 16CPU's, 20 GB RAM and 20GB swap space. We have apache (1.3.27) and tomcat(3.3) in our production servers with JVM (1.4.2 recently upgraded). The frequent problems we face are: I was surprised to see tomcat 3.x and apache 1.x with such massive hardware and elaborate setup. Any particular reason why you don't have more recent versions of tomcat and apache httpd? - High GC (increased pause time when the NewSize=128m -ms=256m -mx=778m and using the default LWP synchronization scheme with GC parameters PrintGCDetails and Time Stamps to analyse in GCportal)(these setting are for individual tomcats) You're doing -Xmx, -Xms, -XX:NewSize, right? Not exactly as you typed above. You should keep tuning to reduce your GC pause: the train (-Xincgc) or parallel collectors would probably be good for you. - The Process running the JVM reached 1GB of size in the 'TOP' list recently, which at this point had to be killed on one of the tomcats. If -Xmx=778m, that's the max heap. 778m is a strange number BTW: how did you arrive at that? Anyways, the output from top represents the whole JVM, i.e. heap + stack + symbols + OS overhead, so it's always more than the heap itself. However, a nearly 25% overhead (778mb-1024mb) is pretty wild. The difference between -Xmx and top when the heap is maxed out on Solaris is usually closer to 12-15%. and sometime with high GC and CPU usage. The 3rd production machine causes high number of thread due to High GC most of the times and on You have your causality mixed up here. High GC does not cause a high number of threads. Once the database connections are high with an increased number of threads, they are unable to come back to the normal condition and we have to kill this tomcat as after some time when the load increases, Why are they unable to come back? Are they not released back into the pool? Do they go bad but not abandoned by the pool? One last thing, there are some occasional tomcat death (JVM crashes) once in a while. What are the crash messages? If they're internal JVM crashes (producing hs_err_[pid] files), make sure you have the latest Solaris patches required for your JDK. There are a bunch of them required for JDK 1.4.x. Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Issues | Suggestion any?
to be done/checked at this stage. Thanks in advance. Regards, Arnab From: Shapira, Yoav [mailto:[EMAIL PROTECTED] Sent: Thu 10/23/2003 6:07 PM To: Tomcat Users List Subject: RE: Issues | Suggestion any? Howdy, The (4) production servers are running Solaris OS with 4CPU, 4GB RAM and 7GB of swap space. In all we have 12 tomcats and 4 Apaches. Each machine is equipped with one apache and 3 tomcats. The Database machines is also running Solaris with 16CPU's, 20 GB RAM and 20GB swap space. We have apache (1.3.27) and tomcat(3.3) in our production servers with JVM (1.4.2 recently upgraded). The frequent problems we face are: I was surprised to see tomcat 3.x and apache 1.x with such massive hardware and elaborate setup. Any particular reason why you don't have more recent versions of tomcat and apache httpd? - High GC (increased pause time when the NewSize=128m -ms=256m -mx=778m and using the default LWP synchronization scheme with GC parameters PrintGCDetails and Time Stamps to analyse in GCportal)(these setting are for individual tomcats) You're doing -Xmx, -Xms, -XX:NewSize, right? Not exactly as you typed above. You should keep tuning to reduce your GC pause: the train (-Xincgc) or parallel collectors would probably be good for you. - The Process running the JVM reached 1GB of size in the 'TOP' list recently, which at this point had to be killed on one of the tomcats. If -Xmx=778m, that's the max heap. 778m is a strange number BTW: how did you arrive at that? Anyways, the output from top represents the whole JVM, i.e. heap + stack + symbols + OS overhead, so it's always more than the heap itself. However, a nearly 25% overhead (778mb-1024mb) is pretty wild. The difference between -Xmx and top when the heap is maxed out on Solaris is usually closer to 12-15%. and sometime with high GC and CPU usage. The 3rd production machine causes high number of thread due to High GC most of the times and on You have your causality mixed up here. High GC does not cause a high number of threads. Once the database connections are high with an increased number of threads, they are unable to come back to the normal condition and we have to kill this tomcat as after some time when the load increases, Why are they unable to come back? Are they not released back into the pool? Do they go bad but not abandoned by the pool? One last thing, there are some occasional tomcat death (JVM crashes) once in a while. What are the crash messages? If they're internal JVM crashes (producing hs_err_[pid] files), make sure you have the latest Solaris patches required for your JDK. There are a bunch of them required for JDK 1.4.x. Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Issues | Suggestion any?
Howdy, I wish your mail program would quote properly -- it's much tougher to read your message without knowing what you wrote and what you quoted ;( Yes we are doing a -Xmx, -Xms, -XX:NewSize and not as I typed, sorry about the confusion. We are in the process to use either of the parallel GC algorithm with jvm 1.4.2 but dont have grounds to prove it would be better but only theoretical (wish if you can point to some). But, we need a parallel collector as we have 4 CPU's per machine and that in fact would help it with some more parameters like compaction. You would never be able to find GC numbers specific to your app. You must benchmark it yourself, on your servers, to get accurate numbers. Only that will tell you which GC settings are best. At the scale you're dealing with, the characteristics of your application matter a whole lot to the GC performance. I don't have at present the idea about this number (778m). But, I was more You should know why you're limited the heap at that unusual number. interested to understand the output that pmap returns and am enclosing one with this mail right now: I'm not a pmap expert, I've never found it useful in debugging performance or GC problems. I'm sure other people on the list could help more in this area. The output marked in red, what does this actually signify ??? Excess swapping ??? Why is the heap size in pmap output not equal to the one under the column size in 'TOP' ??? Top lists a lot more than just the heap, so you can't compare the output from top to just the heap in another tool's output. There was nothing marked in red in your message, so I don't know what you mean by the red question. Anyways what I meant was, if swapping was causing some problems here and do we require more memory or do we need to tune the application more. Comments??? When you look at top, it tells you how much swap space is being used. You want to minimize the amount of swap space used. If you have more physical memory, let the JVM use it by increasing the -Xmx setting. You have your causality mixed up here. High GC does not cause a high number of threads. Yeah I think you are right that high threads don't cause the GC but the default % heap which when filled will invoke the GC. But, more or less all these threads account for database connections actually and to some downloads. That's not what I said: What I said and what you say above are both right, but not the same thing. Why are they unable to come back? Are they not released back into the pool? Do they go bad but not abandoned by the pool? This is what I wanted to know but seems like the connections are held No one can answer that except you. Run your app inside a profiler and see what holds references to the connection objects. The crash message and id was checked. This was found an active bug on the sun's bug database. But, seems they have corrected it in jdk1.4.2. As we have't had any kills all of a sudden yet, it may have been solved or ...?? ... or you've been lucky so far. You should have the latest OS patches for the JDK installed as a matter of proper procedure. Keep watching this, but if it's OK for now, focus more on the memory usage. Please help me find a way out of this or a checklist of what need to be done/checked at this stage. A checklist ;) ;) Amusing. I used to go out with a girl who worked at Sapient. Are you up here in Boston? Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Issues | Suggestion any?
Did you have the same issue with other versions of the JVM? Wade -Original Message- From: Arnab Chakravarty [mailto:[EMAIL PROTECTED] Sent: Thursday, October 23, 2003 2:19 AM To: [EMAIL PROTECTED] Subject: Issues | Suggestion any? Hi all, The application that we run is based on MVC architecture and is built on Jsp, Java and oracle. The (4) production servers are running Solaris OS with 4CPU, 4GB RAM and 7GB of swap space. In all we have 12 tomcats and 4 Apaches. Each machine is equipped with one apache and 3 tomcats. The Database machines is also running Solaris with 16CPU's, 20 GB RAM and 20GB swap space. We have apache (1.3.27) and tomcat(3.3) in our production servers with JVM (1.4.2 recently upgraded). The frequent problems we face are: - High GC (increased pause time when the NewSize=128m -ms=256m -mx=778m and using the default LWP synchronization scheme with GC parameters PrintGCDetails and Time Stamps to analyse in GCportal)(these setting are for individual tomcats) - The Process running the JVM reached 1GB of size in the 'TOP' list recently, which at this point had to be killed on one of the tomcats. - The load balancing scheme in the worker.properties are 4,3,2,1 (Production Machine) 1,2,3,4 4,3,2,1 1,2,3,4 where the load is seen mostly on production machine 4. On this machine the first tomcat spikes up with high threads and Database connections and sometime with high GC and CPU usage. The 3rd production machine causes high number of thread due to High GC most of the times and on this machine the tomcat took around 1GB of memory. The JDBC connections are also high sometime from this machien and thus increased in the pause time causes this tomcat to stop responnding after some time. On running pmap, it was seen 123MB was taken as heap, there were other listing of high memories of 450, 136 and 65 MB in the anon listing. I read in one of the articles that anon was something called anonymous pages and was caused due to swapping. But, does that mean that there is excessive swapping which can cause this GC time to go high because swapping increasing some overhead here of writting this 450MB + other chunks as part of the swap space? Once the database connections are high with an increased number of threads, they are unable to come back to the normal condition and we have to kill this tomcat as after some time when the load increases, apache starts backing up because of tomcat not accepting connections and then the whole scenario is even worse. All tomcat show high threads thereafter. Please help me in case you have any pointers to resolve this scenario. One last thing, there are some occasional tomcat death (JVM crashes) once in a while. Thanks in Advance, Arnab Chakravarty - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Issues | Suggestion any?
Arnab, Once the database connections are high with an increased number of threads, they are unable to come back to the normal condition and we have to kill this tomcat as after some time when the load increases, Why are they unable to come back? Are they not released back into the pool? Do they go bad but not abandoned by the pool? What type of DB connection pooling are you doing? I had similar problems with a client who had a braindead connection pooling system... lost connections (as in, never returned to the pool), deadlock, and crashes when they allowed the pool to grow indefinately; they ran out of memory. Also, why are you running three copies of Tomcat on each machine? Why not just the one VM? Oracle connectors take like 50MB each (on the server-size) to handle, so if you have a pool size of 10 connections for each tomcat instance, and 12 tomcat instances total, then your database machine must allocate over half a gig of memory to handle JDBC connections. I realize that your DB machine is a total *monster*, but I figure that the fewer things running, the easier they are to debug. -chris - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
