Re: Tomcat pausing and no java process
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Miguel, (Bringing this back onto the list) On 6/17/12 5:11 AM, Miguel González Castaños wrote: This weekend I made a test to upgrade from 1.5 32 bits jdk to a 1.7 64 bits jdk. Unfortunately even I increase the heap memory to 1gb, the system gets an outofmemory error after 10-15 minutes of being idle Going from a 32-bit JVM to a 64-bit JVM is likely to *increase* your webapp's memory needs even with the same heap size. The reason is that all pointers (both native and Java references) will be double-wide with respect to a 32-bit JVM. You can run with compressed OOPs but that doesn't cover the JVM itself: you will necessarily require more memory to run a 64-bit JVM. If your webapp runs out of memory after 15 minutes of idle time, then something is terribly wrong. I would take the following steps: 1. Except for -Xmx (and maybe -Xms), remove *all* heap-related settings from your JVM launch command. Your existing settings look like failed attempts to solve a memory leak within the webapp. 2. Launch your webapp in a safe environment (you *do* have an environment identical to production where you can to testing like this, right?) and let it sit idle. Take a thread dump and see if any threads exist that you aren't expecting: that is, something other than a bunch of idle request-processor threads. Maybe your webapp performs some kind of background operation periodically, and that's okay too. Anything you don't recognize: investigate. 3. Take heap dumps at intervals or maybe 5 minutes (that should give you 3 heap dumps before your idle webapp fails and the JVM dies). Load-up those heap-dumps in a heap-analysis tool and look for differences. Maybe you'll find out that you have a million foo.bar.Cache objects being created every minute or something. Obviously investigate those. 4. Once you can get your webapp to be stable during /idle/ time, run a load-test against it (you *do* have a load-test for your webapp, right?) and repeat step #3 above. Using these steps, you ought to be able to identify and fix 90% of memory leaks with very little trouble. Now I have installed a 1.7 32 bits jdk and leave the heap memory to what it was with 1.5 32 bits jdk: Xms128m and Xmx512m. It seems to be stable. Unfortunately this heap size proofed last week not being enough. I analyzed the heap dumps and I have realized there is a process that eats up 200-300 mb. This thread what it does is to load a 5k-10k objects in memory that represent an email and other info that is used to send emails to our clients. From my perspective of an old coder that tried to use memory as less as I could loading such big number of objects to send 200-300 emails an hour doesn't scale. Specially if this company expects to grow in the future and send more emails. Agreed. You might want to separate this email thing into a separate process, so that if/when it takes up a lot of memory and dies, it doesn't bring-down your webapp at the same time. As I see it, increasing the heap is not the solution, but it can be a patch until we get another programmer on board and can refactor the code. Also the owners want to have the impression that the webapp is stable again and feel secure to perform next steps: get a bigger dedicated server (not a virtual one) and get a new programmer/company to do things more professionally. Good plan. Otherwise, you will not have a stable product. Investing in identifying and fixing whatever is wrong is money well-spent. So the question is, how can I increase the heap? Any log that give me any clue? I have separated gc.log to see how the young, perm and old generations numbers evolve. You want to increase the heap? You just need to use -Xmx. You can use jconsole or even jmap to ask the JVM what the effective heap size is once it has been launched. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/faF0ACgkQ9CaO5/Lv0PCM/wCgmYWZb47/twSvlgPn6d2y4Dlq rs0AoK7ouint5uVkx3kaFsRZ8W68uBeK =qSCj -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat pausing and no java process
On 07/05/2012 23:13, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Miguel, On 5/5/12 5:41 AM, Miguel González Castaños wrote: Yes, I have pinpointed several queries that took quite long and specially subqueries. I forwarded this info to the developer but she said it was alright. More than a second to run a query seems to be a lot to me. That depends on what the query does. If it checks permissions to login, that's insane and you should get a new programmer. If it's a reporting query that gathers lots of data then 1 second might not be quite so far-fetched. I have checked directly the slow query log to pinpoint the real queries. I found many subqueries, which I believe is not a good practice, am I right? There's nothing wrong with subqueries in general, it's just that the MySQL query optimizer appears to be particularly stupid when it comes to them, and you often end up with *DEPENDENT* subqueries which essentially issue N queries instead of only 2 queries (if the subquery for instance were to be independent, it could be evaluated first and then essentially substituted in the outer query). Only EXPLAIN can tell you what will happen with each query. Remember to use live data with EXPLAIN because the query optimizer is very sensitive to the data it will be scanning... if you have a test table with 5 records in it, you're going to get worthless EXPLAIN results. Sorry I didn't reply to your email, but somehow it got lost in my inbox. Well, in the end my client managed to get the developer onboard again and to rewrite the subqueries since they were not using any index at all. Let's see how that goes. Thread dumps can be very helpful as well. You might also want to look at running 'sar' all the time to get CPU/IO/etc sampling to see if you have any other kind of resource shortcoming. I have just installed sar yesterday and I'm downloading the reports through ksar remotely through SSH. Nice tool BTW. Tomorrow we are probably getting many clients connecting to the webapp and we can get some conclusions. I have just realized the server is running a 32 bits 1.5 JDK and the OS is a 64 bits linux machine. I'm wondering if using a JVM 1.7 for 64 bits could give us more room to have more people connected simultaneously. Miguel - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat pausing and no java process
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Miguel, On 5/5/12 5:41 AM, Miguel González Castaños wrote: Yes, I have pinpointed several queries that took quite long and specially subqueries. I forwarded this info to the developer but she said it was alright. More than a second to run a query seems to be a lot to me. That depends on what the query does. If it checks permissions to login, that's insane and you should get a new programmer. If it's a reporting query that gathers lots of data then 1 second might not be quite so far-fetched. I have checked directly the slow query log to pinpoint the real queries. I found many subqueries, which I believe is not a good practice, am I right? There's nothing wrong with subqueries in general, it's just that the MySQL query optimizer appears to be particularly stupid when it comes to them, and you often end up with *DEPENDENT* subqueries which essentially issue N queries instead of only 2 queries (if the subquery for instance were to be independent, it could be evaluated first and then essentially substituted in the outer query). Only EXPLAIN can tell you what will happen with each query. Remember to use live data with EXPLAIN because the query optimizer is very sensitive to the data it will be scanning... if you have a test table with 5 records in it, you're going to get worthless EXPLAIN results. Many queries take around 2 seconds and I found two queries that took 16 seconds and 64 seconds to run (with many subqueries nested). Those numbers are meaningless to anyone who isn't on your development team. A 2-second query might be perfectly reasonable... depending on the actual query, the requirements, the frequency of that query being executed, user expectations, etc. Is there any way I can log if any query is causing any trouble in the Tomcat resources? If a query takes 64 seconds to run, it means that a JDBC Connection will be absent from your connection pool for *at least* that long. If you have P pooled connections that take T seconds to execute on average, than you can only serve P/T requests per second. If your webapp is very database-heavy, that can be a problem depending on your values for P and T. Note that simply increasing the number of connections in your pool may make things worse (because you are giving the database more work that it can't keep up with). Sometimes, limiting the number of connections can be beneficial, because the average response time from the database will improve, while the average wait time for a connection from the pool will increase. It all gets down to the resources you have. The Tomcat DBCP (not the default one) has some nice options for instrumentation -- you'll have to read about those on your own because I don't have any experience with them. Tomcat and/or commons-dbcp expose some information via JMX but I think it's fairly basic stuff like average-wait-times for checkout and stuff like that. Something I've done over the years is develop a habit of putting trace logs everywhere. You can always configure the logger on the fly to log some trace output during times where performance starts to nosedive. Thread dumps can be very helpful as well. You might also want to look at running 'sar' all the time to get CPU/IO/etc sampling to see if you have any other kind of resource shortcoming. The developer is quitting so I hope a new developer is more cooperative and we can work out these things. So you're saying that you're not going to get a lot of good information out of him/her? Not much. $0.02: Don't wait to fire him or her: revoke access to everything and get them out the door. If they aren't going to be helpful, at least make sure they aren't going to be harmful. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+oOvQACgkQ9CaO5/Lv0PASbwCfXtxWE+aCAAk4COYToY4twySX szEAoLUdvfr7Qm/SOmdHm1p/NtjoVu+W =4uoI -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat pausing and no java process
Yes, I have pinpointed several queries that took quite long and specially subqueries. I forwarded this info to the developer but she said it was alright. More than a second to run a query seems to be a lot to me. That depends on what the query does. If it checks permissions to login, that's insane and you should get a new programmer. If it's a reporting query that gathers lots of data then 1 second might not be quite so far-fetched. I have checked directly the slow query log to pinpoint the real queries. I found many subqueries, which I believe is not a good practice, am I right? Many queries take around 2 seconds and I found two queries that took 16 seconds and 64 seconds to run (with many subqueries nested). Is there any way I can log if any query is causing any trouble in the Tomcat resources? The problem is that since the database E/R is not documented, it's going to take a while to improve the performance of the queries. Look into SchemaSpy. Nice The developer is quitting so I hope a new developer is more cooperative and we can work out these things. So you're saying that you're not going to get a lot of good information out of him/her? Not much. In the meantime I will check what I can do. I have also downloaded mysqltuner script and mysqlfragfinder a long time ago and follow the output of these tuning scripts. Just rememeber to vet all the recommendations. Don't just read increase your query cache size as a recommendation and blindly apply that rule. Many times the query cache is less useful that you might think. Ok, thanks for the hint. Miguel - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat pausing and no java process
Dear all, My Tomcat 5.5 server reports in catalina.out under heavy load that Tomcat is pausing. I checked and no java process was running and I had to start Tomcat manually. No errors before in catalina.out. The only error found today (It crashed 3 times in 3 hours) was an outofmemory error. I changed from 1024Mb to 1536Mb in catalina.sh: CATALINA_OPTS=$CATALINA_OPTS -Xms1536m -Xmx1536m After installing Javamelody I saw that memory use was reaching 1056 Mb (probably that triggered the outofmemory error). Anything I can check or any setting I can set to get more information in the next crash? Miguel - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat pausing and no java process
On 03/05/2012 11:35, Miguel González Castaños wrote: Dear all, My Tomcat 5.5 server reports in catalina.out under heavy load that Tomcat is pausing. I checked and no java process was running and I had to start Tomcat manually. Can you post an example of the message from catalina.out that says Tomcat is pausing? No errors before in catalina.out. The only error found today (It crashed 3 times in 3 hours) was an outofmemory error. I changed from 1024Mb to 1536Mb in catalina.sh: CATALINA_OPTS=$CATALINA_OPTS -Xms1536m -Xmx1536m Do you actually have 2Gb of free RAM to assign to the Tomcat process? After installing Javamelody I saw that memory use was reaching 1056 Mb (probably that triggered the outofmemory error). Anything I can check or any setting I can set to get more information in the next crash? Enable JMX connect to Tomcat using VisualVM (with the MBean Memory Pool plugins). Monitor thread counts memory usage. Enable an access log, process it to determine traffic volume. Maybe there's a problem, maybe you're just seeing more traffic than your app(s) can handle with a single Tomcat. p -- [key:62590808] signature.asc Description: OpenPGP digital signature
Re: Tomcat pausing and no java process
On 03/05/2012 15:29, Pid wrote: On 03/05/2012 11:35, Miguel González Castaños wrote: Dear all, My Tomcat 5.5 server reports in catalina.out under heavy load that Tomcat is pausing. I checked and no java process was running and I had to start Tomcat manually. Can you post an example of the message from catalina.out that says Tomcat is pausing? May 3, 2012 8:40:10 AM org.apache.catalina.storeconfig.StoreLoader load INFO: Find registry server-registry.xml at classpath resource May 3, 2012 8:40:10 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 5771 ms May 3, 2012 9:56:54 AM org.apache.coyote.http11.Http11BaseProtocol pause INFO: Pausing Coyote HTTP/1.1 on http-80 May 3, 2012 9:56:54 AM org.apache.coyote.http11.Http11BaseProtocol pause INFO: Pausing Coyote HTTP/1.1 on http-443 May 3, 2012 1:35:03 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/bin/jdk1.5.0_17/jre/lib/i386/server:/usr/bin/jdk1.5.0_17/jre/lib/i386:/usr/bin/jdk1.5.0_17/jre/../lib/i386 May 3, 2012 1:35:03 PM org.apache.coyote.http11.Http11BaseProtocol init I have just realized about that APR is missing, can be affecting to this issue? No errors before in catalina.out. The only error found today (It crashed 3 times in 3 hours) was an outofmemory error. I changed from 1024Mb to 1536Mb in catalina.sh: CATALINA_OPTS=$CATALINA_OPTS -Xms1536m -Xmx1536m Do you actually have 2Gb of free RAM to assign to the Tomcat process? Well, the machine has 4 Gb and only runs MySQL and Tomcat. After installing Javamelody I saw that memory use was reaching 1056 Mb (probably that triggered the outofmemory error). Anything I can check or any setting I can set to get more information in the next crash? Enable JMX connect to Tomcat using VisualVM (with the MBean Memory Pool plugins). Monitor thread counts memory usage. I have installed javamelody an the thread count is around 120, reaching a max of 180. Memory usage grows very fast until Tomcat hangs. I suspect is an issue of the garbage collector and setting the heap memory too high. I have reset it to 768 Mb and now the memory seems to keep quite low, some people say that Tomcat 5.x doesn't like much more than 1 Gb of heap memory and that GC is not doing his job: http://serverfault.com/questions/281515/periodic-unresponsiveness-in-tomcat Miguel - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat pausing and no java process
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Miguel, On 5/3/12 6:35 AM, Miguel González Castaños wrote: Anything I can check or any setting I can set to get more information in the next crash? Add -XX:+HeapDumpOnOutOfMemoryError to your CATALINA_OPTS environment variable before you launch Tomcat and you'll get a heap dump file when OOME occurs. You could also enable verbose gc if you want to go back and graph the memory usage.. but if you're using javamelody you probably already have that. Once you have the heap dump, look for either single objects that are taking a huge amount of memory or one particular class that has a lot of objects and/or takes up a lot of memory per object. Follow those objects to their GC roots and you'll find out where the references are. Perhaps you have a memory leak entirely by accident, or maybe a cache that isn't tuned properly for your heap size. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+im5MACgkQ9CaO5/Lv0PD7uQCgiKEtK56xU+l6kw/0UmxruLBf 95MAnRJ3hfZd/POoD09B12MglevNqekk =tzaU -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat pausing and no java process
Hi Miguel, To be more precise about used memory and active threads count, I think that you could send a screenshot of the main graphics of javamelody in your browser (used memory, http sessions, active threads, etc...). And once you have the heap dump as said by Christopher, Eclipse MAT is quite good to analyze the heap dump. Emeric Le 03/05/2012 16:52, Christopher Schultz a écrit : -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Miguel, On 5/3/12 6:35 AM, Miguel González Castaños wrote: Anything I can check or any setting I can set to get more information in the next crash? Add -XX:+HeapDumpOnOutOfMemoryError to your CATALINA_OPTS environment variable before you launch Tomcat and you'll get a heap dump file when OOME occurs. You could also enable verbose gc if you want to go back and graph the memory usage.. but if you're using javamelody you probably already have that. Once you have the heap dump, look for either single objects that are taking a huge amount of memory or one particular class that has a lot of objects and/or takes up a lot of memory per object. Follow those objects to their GC roots and you'll find out where the references are. Perhaps you have a memory leak entirely by accident, or maybe a cache that isn't tuned properly for your heap size. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+im5MACgkQ9CaO5/Lv0PD7uQCgiKEtK56xU+l6kw/0UmxruLBf 95MAnRJ3hfZd/POoD09B12MglevNqekk =tzaU -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat pausing and no java process
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Miguel, On 5/3/12 10:28 AM, Miguel González Castaños wrote: On 03/05/2012 15:29, Pid wrote: On 03/05/2012 11:35, Miguel González Castaños wrote: Dear all, My Tomcat 5.5 server reports in catalina.out under heavy load that Tomcat is pausing. I checked and no java process was running and I had to start Tomcat manually. Can you post an example of the message from catalina.out that says Tomcat is pausing? May 3, 2012 8:40:10 AM org.apache.catalina.storeconfig.StoreLoader load INFO: Find registry server-registry.xml at classpath resource May 3, 2012 8:40:10 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 5771 ms May 3, 2012 9:56:54 AM org.apache.coyote.http11.Http11BaseProtocol pause INFO: Pausing Coyote HTTP/1.1 on http-80 May 3, 2012 9:56:54 AM org.apache.coyote.http11.Http11BaseProtocol pause INFO: Pausing Coyote HTTP/1.1 on http-443 May 3, 2012 1:35:03 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/bin/jdk1.5.0_17/jre/lib/i386/server:/usr/bin/jdk1.5.0_17/jre/lib/i386:/usr/bin/jdk1.5.0_17/jre/../lib/i386 May 3, 2012 1:35:03 PM org.apache.coyote.http11.Http11BaseProtocol init I have just realized about that APR is missing, can be affecting to this issue? Unlikely. If you are using SSL or serving large static files, APR/tcnative should be something that you look into using. Honestly, I would put more effort into upgrading to a more recent version of Tomcat than looking into using APR/tcnative. No errors before in catalina.out. The only error found today (It crashed 3 times in 3 hours) was an outofmemory error. I changed from 1024Mb to 1536Mb in catalina.sh: CATALINA_OPTS=$CATALINA_OPTS -Xms1536m -Xmx1536m Do you actually have 2Gb of free RAM to assign to the Tomcat process? Well, the machine has 4 Gb and only runs MySQL and Tomcat. I hope you have small indexes in MySQL :) I have installed javamelody an the thread count is around 120, reaching a max of 180. Memory usage grows very fast until Tomcat hangs. I suspect is an issue of the garbage collector and setting the heap memory too high. If you suspect that you have set the heap memory too high, why are you increasing it? It really seems like you are just grasping at straws and trying whatever technique comes to mind at the time. Stop. Instrument your webapp. Take data. When you have something substantive, draw a conclusion and confirm it. Only then should you take corrective action. Right now, you are changing all sorts of things and you'll never know what was the one thing that worked because you've changed everything. I have reset it to 768 Mb and now the memory seems to keep quite low, some people say that Tomcat 5.x doesn't like much more than 1 Gb of heap memory and that GC is not doing his job: http://serverfault.com/questions/281515/periodic-unresponsiveness-in-tomcat Malarkey. Tomcat 5.5 doesn't really care what heap size it's got as long as it's more than 4MiB or so. The page you referenced in here doesn't even mention Tomcat 5.5. Usually the GC does do its job properly, and that includes pausing the whole JVM in order to reclaim memory if necessary. The above thread from ServerFault is about unresponsiveness, not about OOME: the GC caused unresponsiveness to avoid OOME (and it worked). Stop making crazy claims and actually look at your own environment to figure out what's going on. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+inVAACgkQ9CaO5/Lv0PB3PQCbBuaYoIurA30z8N3dtLElQXRf MdAAnid8y+qveidJMq5j86946bVYD2Ip =7lpb -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat pausing and no java process
On 03/05/2012 15:59, Christopher Schultz wrote: Miguel, On 5/3/12 10:28 AM, Miguel González Castaños wrote: On 03/05/2012 15:29, Pid wrote: On 03/05/2012 11:35, Miguel González Castaños wrote: Dear all, My Tomcat 5.5 server reports in catalina.out under heavy load that Tomcat is pausing. I checked and no java process was running and I had to start Tomcat manually. Can you post an example of the message from catalina.out that says Tomcat is pausing? May 3, 2012 8:40:10 AM org.apache.catalina.storeconfig.StoreLoader load INFO: Find registry server-registry.xml at classpath resource May 3, 2012 8:40:10 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 5771 ms May 3, 2012 9:56:54 AM org.apache.coyote.http11.Http11BaseProtocol pause INFO: Pausing Coyote HTTP/1.1 on http-80 May 3, 2012 9:56:54 AM org.apache.coyote.http11.Http11BaseProtocol pause INFO: Pausing Coyote HTTP/1.1 on http-443 May 3, 2012 1:35:03 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/bin/jdk1.5.0_17/jre/lib/i386/server:/usr/bin/jdk1.5.0_17/jre/lib/i386:/usr/bin/jdk1.5.0_17/jre/../lib/i386 May 3, 2012 1:35:03 PM org.apache.coyote.http11.Http11BaseProtocol init I have just realized about that APR is missing, can be affecting to this issue? Unlikely. If you are using SSL or serving large static files, APR/tcnative should be something that you look into using. Honestly, I would put more effort into upgrading to a more recent version of Tomcat than looking into using APR/tcnative. No errors before in catalina.out. The only error found today (It crashed 3 times in 3 hours) was an outofmemory error. I changed from 1024Mb to 1536Mb in catalina.sh: CATALINA_OPTS=$CATALINA_OPTS -Xms1536m -Xmx1536m Do you actually have 2Gb of free RAM to assign to the Tomcat process? Well, the machine has 4 Gb and only runs MySQL and Tomcat. ... and an operating system some other system processes and services. You didn't actually answer the question. I hope you have small indexes in MySQL :) Quite. I have installed javamelody an the thread count is around 120, reaching a max of 180. Memory usage grows very fast until Tomcat hangs. I suspect is an issue of the garbage collector and setting the heap memory too high. If you suspect that you have set the heap memory too high, why are you increasing it? It really seems like you are just grasping at straws and trying whatever technique comes to mind at the time. Stop. Instrument your webapp. Take data. When you have something substantive, draw a conclusion and confirm it. Only then should you take corrective action. Right now, you are changing all sorts of things and you'll never know what was the one thing that worked because you've changed everything. I have reset it to 768 Mb and now the memory seems to keep quite low, some people say that Tomcat 5.x doesn't like much more than 1 Gb of heap memory and that GC is not doing his job: Utter nonsense. http://serverfault.com/questions/281515/periodic-unresponsiveness-in-tomcat Malarkey. +1 Tomcat 5.5 doesn't really care what heap size it's got as long as it's more than 4MiB or so. The page you referenced in here doesn't even mention Tomcat 5.5. Usually the GC does do its job properly, and that includes pausing the whole JVM in order to reclaim memory if necessary. The above thread from ServerFault is about unresponsiveness, not about OOME: the GC caused unresponsiveness to avoid OOME (and it worked). Stop making crazy claims and actually look at your own environment to figure out what's going on. +1 p -- [key:62590808] signature.asc Description: OpenPGP digital signature
Re: Tomcat pausing and no java process
Unlikely. If you are using SSL or serving large static files, APR/tcnative should be something that you look into using. I do use SSL, but all is dynamic jsps Honestly, I would put more effort into upgrading to a more recent version of Tomcat than looking into using APR/tcnative. No errors before in catalina.out. The only error found today (It crashed 3 times in 3 hours) was an outofmemory error. I changed from 1024Mb to 1536Mb in catalina.sh: CATALINA_OPTS=$CATALINA_OPTS -Xms1536m -Xmx1536m Do you actually have 2Gb of free RAM to assign to the Tomcat process? Well, as someone asked: ]# cat /proc/meminfo MemTotal: 4194304 kB MemFree: 3700308 kB Tomcat 5.5 doesn't really care what heap size it's got as long as it's more than 4MiB or so. The page you referenced in here doesn't even mention Tomcat 5.5. Usually the GC does do its job properly, and that includes pausing the whole JVM in order to reclaim memory if necessary. The above thread from ServerFault is about unresponsiveness, not about OOME: the GC caused unresponsiveness to avoid OOME (and it worked). Stop making crazy claims and actually look at your own environment to figure out what's going on. We had several crashes today: First - 1 outofmemory error Second- 3 Tomcat pausing Third - 2 or 3 times where catalina.out didn't show any error and tomcat died, only a big increase of memory use right before the java process died. That's why I assumed that maybe I have set the heap memory too high (I set it up because of the out of memory errors). That's why I googled to see if anyone had seen issues and that's why I reached that URL I have set up this: -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -Xms128m -Xmx768m -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails I hope this way I can trace if there is any memory leak in the webapp. Considering that the developer decided to restart Tomcat once a week because the webapp didn't have enough resources, I find this very likely. Thanks for your inputs, Miguel - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat pausing and no java process
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Miguel, On 5/3/12 11:37 AM, Miguel González Castaños wrote: We had several crashes today: First - 1 outofmemory error Second- 3 Tomcat pausing Do you mean that Tomcat actually paused its connectors, or do you mean that your site became unresponsive? Were you able to capture a thread dump during this unresponsive period? Did the JVM recover or did you have to kill it and restart. If you had to kill it, that's called beign locked-up and not just pausing for a while. If you can't take a thread dump during these pauses, you'll never know what is happening. Are you using a database? Are you using pooled connections? If so, enable abandoned tracing if that's a possibility for you (Tomcat's two built-in choices of dbcp can do abandoned tracing). Third - 2 or 3 times where catalina.out didn't show any error and tomcat died, only a big increase of memory use right before the java process died. That's why I assumed that maybe I have set the heap memory too high (I set it up because of the out of memory errors). That's why I googled to see if anyone had seen issues and that's why I reached that URL I have set up this: -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -Xms128m -Xmx768m -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails Great, so you've changed heap size *and* GC strategy and you still have no idea what the problem is. We had a performance problem last week that was very severe. One of my friends said well, if you're not using concurrent mark-and-sweep, then that's probably the problem. (I'm not sure of the nuances, but the default GC on my system is ConcurrentMarkSweep, so I don't know if UseConcMarkSweepGC actually gets you anything). I was skeptical. I did my research. The problem was a missing index on a database table. A simple ALTER statement changed 20-second response times back to more reasonable sub-100ms response times. No mucking-around with the JVM required. That's hy I suggest that you do some research *in your own environment* instead of just guessing at random things and changing every setting that you've found via Google. I hope this way I can trace if there is any memory leak in the webapp. If you have a memory leak in your webapp and you have javamelody, I would argue that you should set your heap to the /largest/ you can possibly get away with and then watch it over time. A memory leak is easy to see over time with a big heap. You are limited by a 32-bit JVM and only ~1.5GiB heap limit, so it might be hard. Maybe you've just outgrown your hardware. Considering that the developer decided to restart Tomcat once a week because the webapp didn't have enough resources, I find this very likely. If you have a memory leak, changing the heap size and GC strategy isn't going to help you at all: it will just waste your time. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+isHgACgkQ9CaO5/Lv0PBeJQCePi3qq6ZF8DuuMgooQtnBuLJA NeUAoJ8HChlhmkiUCqkkLAAtY3X8PJKK =jgRd -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat pausing and no java process
On 03/05/2012 18:21, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Miguel, On 5/3/12 11:37 AM, Miguel González Castaños wrote: We had several crashes today: First - 1 outofmemory error Second- 3 Tomcat pausing Do you mean that Tomcat actually paused its connectors, or do you mean that your site became unresponsive? Were you able to capture a thread dump during this unresponsive period? Did the JVM recover or did you have to kill it and restart. If you had to kill it, that's called beign locked-up and not just pausing for a while. Site became unresponsive. When I saw in catalina.out that the connector was pausing and the website was not responsive, I checked the java process and was dead, so I couldn't get a thread dump If you can't take a thread dump during these pauses, you'll never know what is happening. Any setting that can generate a dump automatically as for out of memory errors? Are you using a database? Are you using pooled connections? If so, enable abandoned tracing if that's a possibility for you (Tomcat's two built-in choices of dbcp can do abandoned tracing). Yes, a mysql database and pooled connections. How can I enable that abandoned tracing? I enabled slow queries for mysql, but the developer claims the queries are fine (although some report more than a second to be run). Third - 2 or 3 times where catalina.out didn't show any error and tomcat died, only a big increase of memory use right before the java process died. That's why I assumed that maybe I have set the heap memory too high (I set it up because of the out of memory errors). That's why I googled to see if anyone had seen issues and that's why I reached that URL I have set up this: -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -Xms128m -Xmx768m -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails Great, so you've changed heap size *and* GC strategy and you still have no idea what the problem is. Well, before the heap size was 512m so I have increased it just a little (to overcome the outofmemory errors). Is that increase that much? I can revert the GC strategy. the verbose options for GC are fine? We had a performance problem last week that was very severe. One of my friends said well, if you're not using concurrent mark-and-sweep, then that's probably the problem. (I'm not sure of the nuances, but the default GC on my system is ConcurrentMarkSweep, so I don't know if UseConcMarkSweepGC actually gets you anything). I was skeptical. I did my research. The problem was a missing index on a database table. A simple ALTER statement changed 20-second response times back to more reasonable sub-100ms response times. No mucking-around with the JVM required. I'll retake that path of checking mysql indexes and slow queries. That's hy I suggest that you do some research *in your own environment* instead of just guessing at random things and changing every setting that you've found via Google. I hope this way I can trace if there is any memory leak in the webapp. If you have a memory leak in your webapp and you have javamelody, I would argue that you should set your heap to the /largest/ you can possibly get away with and then watch it over time. A memory leak is easy to see over time with a big heap. You are limited by a 32-bit JVM and only ~1.5GiB heap limit, so it might be hard. Maybe you've just outgrown your hardware. That we have outgrown our hardware, that's probably for sure. This a virtual private server. I have said the owners to move to a dedicated server but I guess they want me to squeeze this VPS as much as I can. Miguel - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat pausing and no java process
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Miguel, On 5/3/12 1:02 PM, Miguel González Castaños wrote: Any setting that can generate a dump automatically as for out of memory errors? Just the -XX:+HeapDumpOnOutOfMemoryError you already have. Tomcat usually doesn't shut down nicely (as evidenced by the pausing of connectors in your log file) when you get an OOME. Are you sure that weekly Tomcat-restart process isn't still running? Are you using a database? Are you using pooled connections? If so, enable abandoned tracing if that's a possibility for you (Tomcat's two built-in choices of dbcp can do abandoned tracing). Yes, a mysql database and pooled connections. How can I enable that abandoned tracing? Check the documentation for DBCP (which Tomcat uses) here: http://commons.apache.org/dbcp/configuration.html I enabled slow queries for mysql, but the developer claims the queries are fine (although some report more than a second to be run). Well, are you actually seeing slow queries in MySQL? It's not good enough to just enable logging... you have to read the log ;) I can revert the GC strategy. the verbose options for GC are fine? Sure: that's just logging. You will have a miniscule performance degradation due to the logging, but I'm sure it's worth it to see what is happening. I'll retake that path of checking mysql indexes and slow queries. Just use mysqldumpslow to be sure. Also, if you suspect a query, use EXPLAIN on it. Mysqldumpslow be default anonymizes queries so you can see a class of equivalent queries that are failing. MySQL determines its query plan based not just upon the structure of the query, but also upon the data and the indexes that are available at the time of the query. You might want to disable this anonymizing of the dat to get an /actual/ query that was slow. Then run EXPLAIN on that to see what the plan was: maybe MySQL is making a bad decision, or maybe you need an additional index. Or maybe the query is just written poorly in the first place. MySQL is notorious for using DEPENDENT SUBQUERIES (awful performance) when such a query is not necessary. That we have outgrown our hardware, that's probably for sure. This a virtual private server. I have said the owners to move to a dedicated server but I guess they want me to squeeze this VPS as much as I can. Sheesh, if it's virtualized, just have them provision a larger instance for you and move the virtual machine. I'd recommend a 64-bit OS/JVM so you can get heaps bigger than about 1.5GiB. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+ix3gACgkQ9CaO5/Lv0PA4mQCgnVETbX5COEfE7smzcYshFpQF of0An0xRGmakspqFsTIcobjonz5QP81U =F8mw -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat pausing and no java process
Just the -XX:+HeapDumpOnOutOfMemoryError you already have. Tomcat usually doesn't shut down nicely (as evidenced by the pausing of connectors in your log file) when you get an OOME. Are you sure that weekly Tomcat-restart process isn't still running? Yes, it's still running. Should i disable it? Are you using a database? Are you using pooled connections? If so, enable abandoned tracing if that's a possibility for you (Tomcat's two built-in choices of dbcp can do abandoned tracing). Yes, a mysql database and pooled connections. How can I enable that abandoned tracing? Check the documentation for DBCP (which Tomcat uses) here: http://commons.apache.org/dbcp/configuration.html I have set logAbandoned=true. I haven't enabled the removeAbandoned and the removeAbandonedTimeout just in case. Will I get the info in the catalina log? I enabled slow queries for mysql, but the developer claims the queries are fine (although some report more than a second to be run). Well, are you actually seeing slow queries in MySQL? It's not good enough to just enable logging... you have to read the log ;) I can revert the GC strategy. the verbose options for GC are fine? Sure: that's just logging. You will have a miniscule performance degradation due to the logging, but I'm sure it's worth it to see what is happening. I'll retake that path of checking mysql indexes and slow queries. Just use mysqldumpslow to be sure. Also, if you suspect a query, use EXPLAIN on it. Mysqldumpslow be default anonymizes queries so you can see a class of equivalent queries that are failing. MySQL determines its query plan based not just upon the structure of the query, but also upon the data and the indexes that are available at the time of the query. You might want to disable this anonymizing of the dat to get an /actual/ query that was slow. Then run EXPLAIN on that to see what the plan was: maybe MySQL is making a bad decision, or maybe you need an additional index. Or maybe the query is just written poorly in the first place. MySQL is notorious for using DEPENDENT SUBQUERIES (awful performance) when such a query is not necessary. I'll check about the anonymizing thing. Right now I get the following cronjob forwarded to my email /usr/bin/mysqldumpslow -s -t /var/log/mysql-slow.log | mail -s Yes, I have pinpointed several queries that took quite long and specially subqueries. I forwarded this info to the developer but she said it was alright. More than a second to run a query seems to be a lot to me. The problem is that since the database E/R is not documented, it's going to take a while to improve the performance of the queries. The developer is quitting so I hope a new developer is more cooperative and we can work out these things. In the meantime I will check what I can do. I have also downloaded mysqltuner script and mysqlfragfinder a long time ago and follow the output of these tuning scripts. That we have outgrown our hardware, that's probably for sure. This a virtual private server. I have said the owners to move to a dedicated server but I guess they want me to squeeze this VPS as much as I can. Sheesh, if it's virtualized, just have them provision a larger instance for you and move the virtual machine. I'd recommend a 64-bit OS/JVM so you can get heaps bigger than about 1.5GiB. :-) I'm trying to convince them. Many thanks for your inputs, you have been very helpful Miguel - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat pausing and no java process
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Miguel, On 5/3/12 2:25 PM, Miguel González Castaños wrote: Just the -XX:+HeapDumpOnOutOfMemoryError you already have. Tomcat usually doesn't shut down nicely (as evidenced by the pausing of connectors in your log file) when you get an OOME. Are you sure that weekly Tomcat-restart process isn't still running? Yes, it's still running. Should i disable it? Well, if it's shutting-down your JVM while you're trying to observe it, then maybe you don't want it doing that. *shrug* I'm just saying that Tomcat doesn't usually just decide to shut down for no good reason. I have set logAbandoned=true. I haven't enabled the removeAbandoned and the removeAbandonedTimeout just in case. Will I get the info in the catalina log? Yes. logAbandoned IIRC logs to System.out, so you'll get that stuff in catalina.out. I also recommend setting logAbandonedTimeout to something reasonable. I don't remember the default. Right now I get the following cronjob forwarded to my email /usr/bin/mysqldumpslow -s -t /var/log/mysql-slow.log | mail -s Whoever wrote that didn't read the mysqldump manual. Who knows *what* that's going to give you. Yes, I have pinpointed several queries that took quite long and specially subqueries. I forwarded this info to the developer but she said it was alright. More than a second to run a query seems to be a lot to me. That depends on what the query does. If it checks permissions to login, that's insane and you should get a new programmer. If it's a reporting query that gathers lots of data then 1 second might not be quite so far-fetched. The problem is that since the database E/R is not documented, it's going to take a while to improve the performance of the queries. Look into SchemaSpy. The developer is quitting so I hope a new developer is more cooperative and we can work out these things. So you're saying that you're not going to get a lot of good information out of him/her? In the meantime I will check what I can do. I have also downloaded mysqltuner script and mysqlfragfinder a long time ago and follow the output of these tuning scripts. Just rememeber to vet all the recommendations. Don't just read increase your query cache size as a recommendation and blindly apply that rule. Many times the query cache is less useful that you might think. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+i5OQACgkQ9CaO5/Lv0PA/GwCgrrZL44k33jhm1AG4rPOefy3n aFgAoKm3mqmr3bvOY1sTyogSPunPL4kt =gl93 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
