RE: How to monitor performance of tomcat
There is 1 finding. We normally monitor the tomcat port using jconsole as- jconsole 10.101.17.79:8891 As per my initial query, our application hangs and we need to restart JBoss Tomcat. I have observed that during this time the port 8891 does not respond as the command, jconsole 10.101.17.79:8891 does not give any console for monitoring. Requesting an update so that the root cause analysis of this issue can be found out. Regards -Original Message- From: Shanti Suresh [mailto:sha...@umich.edu] Sent: Saturday, April 12, 2014 1:12 AM To: Tomcat Users List Subject: Re: How to monitor performance of tomcat On Fri, Apr 11, 2014 at 3:04 PM, André Warnier a...@ice-sa.com wrote: Shanti Suresh wrote: Hi Chris, On Fri, Apr 11, 2014 at 12:17 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Shanti, On 4/11/14, 9:01 AM, Shanti Suresh wrote: Thank you! Great presentation and most wonderful notes! One question - on slide #48, where the notes say, You can see here that the current usage is about 100MiB, less than the 115MiB threshold we set, where is the 100MB or thereabouts shown? Is it the committed value? I don't follow that statement. The statement refers to the used value. - -chris Thank you! I got it. The used value says used 114510568. I was looking for a value closer to 100MiB. 114510568 (~ 109 MB) - 104857600(100 MB) === 9652968 (~ 9 MB) How much closer were you looking for ? Well, I don't know, maybe 1 or 2 MiB over? The threshold is 115 MiB, so if the notes had said the current usage is less than the 115MiB threshold we set, I might not have had any doubt. So in the slide, you were making a point of the current usage being less than the threshold, basically, if I am not mistaken. Thanks, -Shanti Thanks, -Shanti -- *STL Disclaimer:* The content of this message may be legally privileged and confidential and are for the use of the intended recipient(s) only. It should not be read, copied and used by anyone other than the intended recipient(s). If you have received this message in error, please immediately notify the sender, preserve its confidentiality and delete it. Before opening any attachments please check them for viruses and defects. No employee or agent is authorised to conclude any binding agreement on behalf of Sterlite Technologies Limited with another party by email without express written confirmation by authorised person. Visit us at www.sterlitetechnologies.com Please consider environment before printing this email ! - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
[ANN] Apache Tomcat Native 1.1.30 released
The Apache Tomcat team announces the immediate availability of Apache Tomcat Native 1.1.30 stable. Please refer to the change log for the list of changes: http://tomcat.apache.org/native-doc/miscellaneous/changelog.html Downloads: http://tomcat.apache.org/download-native.cgi The Apache Tomcat Native Library provides portable API for features not found in contemporary JDK's. It uses Apache Portable Runtime as operating system abstraction layer and OpenSSL for SSL networking and allows optimal performance in production environments. Thank you, -- The Apache Tomcat Team - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
[ANN] Apache Tomcat Connectors 1.2.40 released
The Apache Tomcat Project is proud to announce the release of version 1.2.40 of Apache Tomcat Connectors. This version fixes few bugs found in previous releases. Full details of these changes and new features, are available in the Apache Tomcat Connectors changelog: http://tomcat.apache.org/connectors-doc/miscellaneous/changelog.html Downloads: http://tomcat.apache.org/download-connectors.cgi Thank you, -- The Apache Tomcat Team - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: AW: AW: tomcat-connectors-1.2.39-windows-x86_64-iis does not work
Hi all, the new tomcat-connectors-1.2.40-windows-x86_64-iis works with the localhost setting in workers.properties! Great! Many thanks to the developers! :-) Regards, Jessica smime.p7s Description: S/MIME cryptographic signature
Re: How to monitor performance of tomcat
Hi Randhir, Have you considered taking a thread dump of the JVM processes, I forget? http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F If you take multiple thread dumps, say, 6, a minute apart, then you may open these up in a thread dump analyzer such as TDA or Samurai and see what threads are deadlocked, or how they are progressing etc. Then restart Tomcats to fix problem. Then analyze the thread dumps post-restart. I find thread and heap dumps useful in addition to monitoring metrics. Thanks, -Shanti On Tue, Apr 15, 2014 at 6:49 AM, Randhir Singh randhir.si...@sterlite.comwrote: There is 1 finding. We normally monitor the tomcat port using jconsole as- jconsole 10.101.17.79:8891 As per my initial query, our application hangs and we need to restart JBoss Tomcat. I have observed that during this time the port 8891 does not respond as the command, jconsole 10.101.17.79:8891 does not give any console for monitoring. Requesting an update so that the root cause analysis of this issue can be found out. Regards -Original Message- From: Shanti Suresh [mailto:sha...@umich.edu] Sent: Saturday, April 12, 2014 1:12 AM To: Tomcat Users List Subject: Re: How to monitor performance of tomcat On Fri, Apr 11, 2014 at 3:04 PM, André Warnier a...@ice-sa.com wrote: Shanti Suresh wrote: Hi Chris, On Fri, Apr 11, 2014 at 12:17 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Shanti, On 4/11/14, 9:01 AM, Shanti Suresh wrote: Thank you! Great presentation and most wonderful notes! One question - on slide #48, where the notes say, You can see here that the current usage is about 100MiB, less than the 115MiB threshold we set, where is the 100MB or thereabouts shown? Is it the committed value? I don't follow that statement. The statement refers to the used value. - -chris Thank you! I got it. The used value says used 114510568. I was looking for a value closer to 100MiB. 114510568 (~ 109 MB) - 104857600(100 MB) === 9652968 (~ 9 MB) How much closer were you looking for ? Well, I don't know, maybe 1 or 2 MiB over? The threshold is 115 MiB, so if the notes had said the current usage is less than the 115MiB threshold we set, I might not have had any doubt. So in the slide, you were making a point of the current usage being less than the threshold, basically, if I am not mistaken. Thanks, -Shanti Thanks, -Shanti -- *STL Disclaimer:* The content of this message may be legally privileged and confidential and are for the use of the intended recipient(s) only. It should not be read, copied and used by anyone other than the intended recipient(s). If you have received this message in error, please immediately notify the sender, preserve its confidentiality and delete it. Before opening any attachments please check them for viruses and defects. No employee or agent is authorised to conclude any binding agreement on behalf of Sterlite Technologies Limited with another party by email without express written confirmation by authorised person. Visit us at www.sterlitetechnologies.com Please consider environment before printing this email ! - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: AW: tomcat-connectors-1.2.39-windows-x86_64-iis does not work
On 04/15/2014 02:58 PM, Alten, Jessica-Aileen wrote: Hi all, the new tomcat-connectors-1.2.40-windows-x86_64-iis works with the localhost setting in workers.properties! Great! Many thanks to the developers! :-) You're welcome. Thanks for filing the bug and confirming the fix. It was that kind of bug that does not show on all boxes and all the time. Regards -- ^TM - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Performance - Java Profiler, JVM instrmentation
Greetings, Chris' presentation on monitoring Tomcat is really nice work. I found that quite useful. Taking it one step further, could I request some recommendations on how we might profile Java code running inside Tomcat? Often, I am stuck with finding out why an application is slow. It could be a consistent, progressive or a sudden problem. These applications do not expose metrics via MBeans. Say, for e.g., a vendor application which has been heavily customized in-house. Some metrics that I find useful during these times are things like concurrent invocations, stall counts on components, call-stack, response-rate etc. Java Melody has a nice built-in dashboard of metrics. Co-relating metrics like that is powerful and helps isolate relatively easy problems. I find that the metrics skim the surface of more involved problems. In Tomcat, is there a way to go deeper into the performance of the code for root-cause analysis and isolate a section of the code or a flow in the code for troubleshooting? How would one go about getting to that place? Let's say, there is no budget for purchasing tools in that space. I find Chris' example on writing filters to map to URL patterns for response-time metrics relevant. I would also like stall counts, concurrent invocations etc. Greatly appreciate your thoughts and opinions. Thanks, -Shanti
Re: How to monitor performance of tomcat
2014-04-15 14:49 GMT+04:00 Randhir Singh randhir.si...@sterlite.com: There is 1 finding. We normally monitor the tomcat port using jconsole as- jconsole 10.101.17.79:8891 As per my initial query, our application hangs and we need to restart JBoss Tomcat. I have observed that during this time the port 8891 does not respond as the command, jconsole 10.101.17.79:8891 does not give any console for monitoring. Requesting an update so that the root cause analysis of this issue can be found out. An often cause for such behaviour is an out-of-memory condition. If you encountered OutOfMemoryError: PermGen error, then JVM will not be able to load any new classes, with obvious fatal consequences. Did you monitor the PermGen memory pool size, or just memory as a whole? If an OOME causes a thread death, it is usually not logged to logging frameworks, but a message in written to System.err (by ThreadGroup,uncaughtException(..)). 2014-04-15 17:40 GMT+04:00 Shanti Suresh sha...@umich.edu: Have you considered taking a thread dump of the JVM processes, I forget? +1 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance - Java Profiler, JVM instrmentation
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Shanti, On 4/15/14, 10:51 AM, Shanti Suresh wrote: Taking it one step further, could I request some recommendations on how we might profile Java code running inside Tomcat? Often, I am stuck with finding out why an application is slow. It could be a consistent, progressive or a sudden problem. These applications do not expose metrics via MBeans. Say, for e.g., a vendor application which has been heavily customized in-house. You need to use a Profiler for that. There are a number of fine Profilers available for Java. I use YourKit because they give free licenses to ASF committers. In Tomcat, is there a way to go deeper into the performance of the code for root-cause analysis and isolate a section of the code or a flow in the code for troubleshooting? How would one go about getting to that place? Let's say, there is no budget for purchasing tools in that space. Tomcat provides no such tools. You can use jconsole for a bit of profiling, but I've never used it heavily. Honestly, if there is no budget for such tools, then there is no budget for improving the performance of your application. You'll have to convince someone with a higher level of clearance that customer needs are worth spending money. I find Chris' example on writing filters to map to URL patterns for response-time metrics relevant. I would also like stall counts, concurrent invocations etc. What is a stall-count? How would you record concurrent invocations, etc.? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTTXM7AAoJEBzwKT+lPKRY/g8P/3hdbqYf5I2s0LaxiHexJ27I tTouN/Gk0+kmOapdoPUxXECC8q/LF51I7aCF5L2D/XY5Jgu5Vx5grwvoZWVdAS5N OB54gLvOq1toVSGTI7C7POqg8S7gqdj1Ec5BCrxiI0HGmSWNYcDQoFNWXL+srE6L xes6o1Kfhlvs/cjmhWagooaGvaNY1Wu7DyhJowaTmWzDecvEmIS+A3ysuSoB+g4B Myc61ZR4Imul6+0iRm59dWjE5ID9cqK9jr5NzFUIcy2zJaCIkNV1SKnliWkZBgp/ b/B7pUWfTnjUiX4lEDjH9WYL6cXO/fkwL70yue1iki/u+MqOJn7c+Qyo2VzRtn1W iYHpm5BKngNER8r/4AhzV7ZvqGlmqOoLHqmh88Seiq8AAcloLdDkXWhR/GQzHkuk +ESCRv5qUmmDwOdk1wcuMjGOk10+tN5zc2ZfrJlchYxVKvPaxHFcFVSczlcb+Bok heetKDK2PZlYlzQ0iACwBHLzmaU/Jycmv3+zVOucBmyeZZBUX6GpAjG2lsh0QPlG DhIPj1PHuC6OYsIGbVeLFirFRo/hS+XwACaY/qBVyyGd9k6NRjtpQ9RMQvb2dUGu CPh52SBzgyOGGu9dZ1EaYSC7Vl21DySDmWr8eUMxIjEYinr0rc5Mv1VqOQEWaWy9 rixRAxhweuazzCqGnat7 =0i+1 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat logging with Log4j
Hi all, We need to add log rotation and log size management to tomcat 7. Tried converting to Log4j steps from tomcat website (http://tomcat.apache.org/tomcat-7.0-doc/logging.html) but did not work, was able to get it to work from (http://mrhaki.blogspot.com/2011/02/configure-log4j-on-tomcat.html). It seems we are not getting any logging from our webapp though, and prior to this change it was getting logged in stdout and stderr. Does something need to be changed in log4j.properties to still capture stdout and stderr to log file with Log4j? We are running Tomcat as a windows service on 2008 R2 with Java 7. Thank you! Scott DISCLAIMER: This electronic mail message and any attached files contain information intended for the exclusive use of the intended addressee and may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify sender if you are an unintended recipient and delete the original message without making copies. Thank you.
Re: Please help diagnosing a random production Tomcat 7.0.53 Internal Server Error!
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Ian, On 4/15/14, 2:52 PM, Ian Long wrote: I need some help from all the tomcat experts out there! I am using tomcat behind apache httpd using mod_jk (1.2.39). About 50-100 times per day (out of many requests), I’m getting an internal server error from Tomcat (error 500), without any exceptions in my code, nor in Tomcat logs. I am only seeing the error in my New Relic application monitoring tool, and I can see them in the mod_jk logs if I turn on debug. As much fun as reading debug logs is, I wasn't able to find a problem in what you posted. Can you maybe highlight the section that indicates a problem? You also didn't post the exception from the Java side. My server is not heavily loaded, with a load average hovering around 0.5 on a 4 cpu system. How many httpd processes are serving this Tomcat? Do you have a mismatch between the number of connections coming from httpd and the number of connections available on the Tomcat side (Connector)? You can see the internal error below at 13:59:13.790. Yes, we can see that there was an error, but not what the error was. My worker setup is very simple: worker.list=worker1 worker.worker1.port=8009 worker.worker1.host=127.0.0.1 worker.worker1.type=ajp13 worker.worker1.connection_pool_timeout=600 worker.worker1.connect_timeout=1 My Connector is also straightforward: Connector port=8009 connectionTimeout=60 minSpareThreads=5 address=127.0.0.1 URIEncoding=UTF-8 enableLookups=false disableUploadTimeout=true maxSpareThreads=75 maxThreads=800 protocol=AJP/1.3 / That all looks okay to me on the face of it. Just a note: you may want to use an Executor for better control of the thread pool. What connector are you actually using Is 800 threads enough to handle whatever might be coming from httpd (or all of your httpd instances)? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTTYSWAAoJEBzwKT+lPKRYzAkP/1Eeusa0Jh6uFoUFg0+wq/cO IF8F0DkQXZ5d1WrYtF3nRhcNXclgfl6kYpNyz9dSN55Sk2hrWFZcSGZnMu4ZZvQE jMg+555tPm36QmmAw3NzUm6wwTpcByjsZuj10fsigiaNW3ucAc2vsQ40ETH5LH+/ E4crD6PFBSfNe5qcF51T6qcPVMMaXxjd5aBWRBfT2sUEogRg3o5Xm6zal+fwQrfR v4mbvwC4bz7ysCXGZQxSh7qQrorpXePIqCrUekAXxPRGxGXbUvj8+alVjY7p0Him w5WyyzbEqIymrARoip/+Xd1nRe7bWdt0sUBqBsKn7KKvUVvvIMbKmtAn398zcP9k l9746MuX0Z9JGuCNDeX/giaUeijckjyY2WjxWY/mU9v75v02jqpPlgzZZhELKv/3 ScE13HgxzPHAiDNXHJuQsJL8HxRbtl29aPV+406kQbolzfMudxXPU2hSIi8MDiYn hTJSZwp47bQngD9Ym8v+EdeExvRg2xLhlIuJc5j+34E9J5R9p/QC7Ru6YyzpESO5 olTzG/5Dt4V75q7mRkMtNiIWku9Ur5dtD+wjLAcQPmcoUuN0pX+rl2L4a7Wp+mqO rCuEZK5Y1S6/DBlu7UcBALe/T0OG8nzld4xLKZJR/oluuQSRXlRw6div4DaoqQMf 4PjjoG0+Hj2KS2aQm/JQ =oqiw -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat logging with Log4j
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Scott, On 4/15/14, 2:50 PM, Scott Bailey wrote: We need to add log rotation and log size management to tomcat 7. Tried converting to Log4j steps from tomcat website (http://tomcat.apache.org/tomcat-7.0-doc/logging.html) but did not work, was able to get it to work from (http://mrhaki.blogspot.com/2011/02/configure-log4j-on-tomcat.html). It seems we are not getting any logging from our webapp though, and prior to this change it was getting logged in stdout and stderr. What steps did you actually take? Does something need to be changed in log4j.properties to still capture stdout and stderr to log file with Log4j? How are you logging from within your application? ServletContext.log()? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTTYbNAAoJEBzwKT+lPKRYHB4P/RUlDHyT2wQTt41S0YB/VbCR leQxOtDJYeO3WZ3qERNP1yzmX3GW2xKfxy8+5yXAgmPugO4LXbW2sj7a4VEDuR3D l5a/AkjqvsEdJFvWAB/77NFzWvcbOfiBT5Iw8AdkoprzaitdsD7UelLA7OlLbSCr EnLp1ZQmVJHEdAaFc0Rr2tH7SY1oSFZM19wmihBPTFZsOfnssiEnDnO6zhxgl9kG IlvNNuadefd0TxUiaNsncNYQatGjNsWHsjf6miYcEuJ2ZEz8x0YBzZv60MP0qiVi U/YvkvQCwGJU9wYvK9SjKpmJrunnB2dt3zKL724+qCw4D8h7qcccq6yytBXGRDha x4847oIbkvG7fqtljQTjfefuh8fFKalEVR8LP3huQtHqkpT4YRnRhJifFwZQmz1E aj7h/dz91F5pWmN673Fs8aO5LAM8qvsk7sT89QtBle0REeFZFEGPqGwBsDDV4sgE Shnr4JuR+xVyGCInHzV/zMDJ4EtVZVHRNFUnG9zS9Q5+FWfiOVc4cqxb4s6NgRKg mVhC8qharbzt/3I0nLNtMIxbz8c7hB2zNGBgjaQHEZ8BNp0+p3MtEOz9fGywi0pR CEL+HtjWbfaV+O8d3HSLyQMqtT5/ZOm10ZBG0s1yFjhaoxwz3yUjLLu2G3TeGJeD rRzHvGKrkAwWRehs+Zgm =g0p8 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Please help diagnosing a random production Tomcat 7.0.53 Internal Server Error!
Thanks for the reply. It looks to me like tomcat just gave up partway through generating the request, I’m trying to figure out why. There are no exceptions in either my application logs or the tomcat log itself, which is frustrating. Thanks, I’ll look into the executor. Apache matches what is set in my connector: IfModule prefork.c StartServers 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 800 MaxClients 800 MaxRequestsPerChild 0 /IfModule Yes, the connector settings should be fine, there are usually less than 20 httpds. Cheers, Ian On April 15, 2014 at 3:13:08 PM, Christopher Schultz (ch...@christopherschultz.net) wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Ian, On 4/15/14, 2:52 PM, Ian Long wrote: I need some help from all the tomcat experts out there! I am using tomcat behind apache httpd using mod_jk (1.2.39). About 50-100 times per day (out of many requests), I’m getting an internal server error from Tomcat (error 500), without any exceptions in my code, nor in Tomcat logs. I am only seeing the error in my New Relic application monitoring tool, and I can see them in the mod_jk logs if I turn on debug. As much fun as reading debug logs is, I wasn't able to find a problem in what you posted. Can you maybe highlight the section that indicates a problem? You also didn't post the exception from the Java side. My server is not heavily loaded, with a load average hovering around 0.5 on a 4 cpu system. How many httpd processes are serving this Tomcat? Do you have a mismatch between the number of connections coming from httpd and the number of connections available on the Tomcat side (Connector)? You can see the internal error below at 13:59:13.790. Yes, we can see that there was an error, but not what the error was. My worker setup is very simple: worker.list=worker1 worker.worker1.port=8009 worker.worker1.host=127.0.0.1 worker.worker1.type=ajp13 worker.worker1.connection_pool_timeout=600 worker.worker1.connect_timeout=1 My Connector is also straightforward: Connector port=8009 connectionTimeout=60 minSpareThreads=5 address=127.0.0.1 URIEncoding=UTF-8 enableLookups=false disableUploadTimeout=true maxSpareThreads=75 maxThreads=800 protocol=AJP/1.3 / That all looks okay to me on the face of it. Just a note: you may want to use an Executor for better control of the thread pool. What connector are you actually using Is 800 threads enough to handle whatever might be coming from httpd (or all of your httpd instances)? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTTYSWAAoJEBzwKT+lPKRYzAkP/1Eeusa0Jh6uFoUFg0+wq/cO IF8F0DkQXZ5d1WrYtF3nRhcNXclgfl6kYpNyz9dSN55Sk2hrWFZcSGZnMu4ZZvQE jMg+555tPm36QmmAw3NzUm6wwTpcByjsZuj10fsigiaNW3ucAc2vsQ40ETH5LH+/ E4crD6PFBSfNe5qcF51T6qcPVMMaXxjd5aBWRBfT2sUEogRg3o5Xm6zal+fwQrfR v4mbvwC4bz7ysCXGZQxSh7qQrorpXePIqCrUekAXxPRGxGXbUvj8+alVjY7p0Him w5WyyzbEqIymrARoip/+Xd1nRe7bWdt0sUBqBsKn7KKvUVvvIMbKmtAn398zcP9k l9746MuX0Z9JGuCNDeX/giaUeijckjyY2WjxWY/mU9v75v02jqpPlgzZZhELKv/3 ScE13HgxzPHAiDNXHJuQsJL8HxRbtl29aPV+406kQbolzfMudxXPU2hSIi8MDiYn hTJSZwp47bQngD9Ym8v+EdeExvRg2xLhlIuJc5j+34E9J5R9p/QC7Ru6YyzpESO5 olTzG/5Dt4V75q7mRkMtNiIWku9Ur5dtD+wjLAcQPmcoUuN0pX+rl2L4a7Wp+mqO rCuEZK5Y1S6/DBlu7UcBALe/T0OG8nzld4xLKZJR/oluuQSRXlRw6div4DaoqQMf 4PjjoG0+Hj2KS2aQm/JQ =oqiw -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance - Java Profiler, JVM instrmentation
On Tue, Apr 15, 2014 at 4:51 PM, Shanti Suresh sha...@umich.edu wrote: Greetings, Hello Shanti, Chris' presentation on monitoring Tomcat is really nice work. I found that quite useful. Taking it one step further, could I request some recommendations on how we might profile Java code running inside Tomcat? Often, I am stuck with finding out why an application is slow. It could be a consistent, progressive or a sudden problem. These applications do not expose metrics via MBeans. Say, for e.g., a vendor application which has been heavily customized in-house. Some metrics that I find useful during these times are things like concurrent invocations, stall counts on components, call-stack, response-rate etc. Java Melody has a nice built-in dashboard of metrics. Co-relating metrics like that is powerful and helps isolate relatively easy problems. I find that the metrics skim the surface of more involved problems. In Tomcat, is there a way to go deeper into the performance of the code for root-cause analysis and isolate a section of the code or a flow in the code for troubleshooting? How would one go about getting to that place? Let's say, there is no budget for purchasing tools in that space. I find Chris' example on writing filters to map to URL patterns for response-time metrics relevant. I would also like stall counts, concurrent invocations etc. There are tools that are doing exactly that for about 7 years out now. You can go to http://newrelic.com and get it for as much as 150 USD per server. Or you can get all the same for free from http://www.moskito.org. And more. regards Leon Greatly appreciate your thoughts and opinions. Thanks, -Shanti
Re: Performance - Java Profiler, JVM instrmentation
On Tue, Apr 15, 2014 at 7:58 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Shanti, On 4/15/14, 10:51 AM, Shanti Suresh wrote: Taking it one step further, could I request some recommendations on how we might profile Java code running inside Tomcat? Often, I am stuck with finding out why an application is slow. It could be a consistent, progressive or a sudden problem. These applications do not expose metrics via MBeans. Say, for e.g., a vendor application which has been heavily customized in-house. You need to use a Profiler for that. There are a number of fine Profilers available for Java. I use YourKit because they give free licenses to ASF committers. Hello Chris, et al, last time I tried to use a profile on a production site it killed it within a second. Usually the performance overhead of a profiler is so huge, that you have no chance to run it in production. And real problems do not occur in test labs ;-) Leon
Re: Please help diagnosing a random production Tomcat 7.0.53 Internal Server Error!
Forgot to mention that it looks like tomcat returned around 50% of what the page should have been, before it hit the Internal Server Error. Cheers, Ian On April 15, 2014 at 3:13:08 PM, Christopher Schultz (ch...@christopherschultz.net) wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Ian, On 4/15/14, 2:52 PM, Ian Long wrote: I need some help from all the tomcat experts out there! I am using tomcat behind apache httpd using mod_jk (1.2.39). About 50-100 times per day (out of many requests), I’m getting an internal server error from Tomcat (error 500), without any exceptions in my code, nor in Tomcat logs. I am only seeing the error in my New Relic application monitoring tool, and I can see them in the mod_jk logs if I turn on debug. As much fun as reading debug logs is, I wasn't able to find a problem in what you posted. Can you maybe highlight the section that indicates a problem? You also didn't post the exception from the Java side. My server is not heavily loaded, with a load average hovering around 0.5 on a 4 cpu system. How many httpd processes are serving this Tomcat? Do you have a mismatch between the number of connections coming from httpd and the number of connections available on the Tomcat side (Connector)? You can see the internal error below at 13:59:13.790. Yes, we can see that there was an error, but not what the error was. My worker setup is very simple: worker.list=worker1 worker.worker1.port=8009 worker.worker1.host=127.0.0.1 worker.worker1.type=ajp13 worker.worker1.connection_pool_timeout=600 worker.worker1.connect_timeout=1 My Connector is also straightforward: Connector port=8009 connectionTimeout=60 minSpareThreads=5 address=127.0.0.1 URIEncoding=UTF-8 enableLookups=false disableUploadTimeout=true maxSpareThreads=75 maxThreads=800 protocol=AJP/1.3 / That all looks okay to me on the face of it. Just a note: you may want to use an Executor for better control of the thread pool. What connector are you actually using Is 800 threads enough to handle whatever might be coming from httpd (or all of your httpd instances)? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTTYSWAAoJEBzwKT+lPKRYzAkP/1Eeusa0Jh6uFoUFg0+wq/cO IF8F0DkQXZ5d1WrYtF3nRhcNXclgfl6kYpNyz9dSN55Sk2hrWFZcSGZnMu4ZZvQE jMg+555tPm36QmmAw3NzUm6wwTpcByjsZuj10fsigiaNW3ucAc2vsQ40ETH5LH+/ E4crD6PFBSfNe5qcF51T6qcPVMMaXxjd5aBWRBfT2sUEogRg3o5Xm6zal+fwQrfR v4mbvwC4bz7ysCXGZQxSh7qQrorpXePIqCrUekAXxPRGxGXbUvj8+alVjY7p0Him w5WyyzbEqIymrARoip/+Xd1nRe7bWdt0sUBqBsKn7KKvUVvvIMbKmtAn398zcP9k l9746MuX0Z9JGuCNDeX/giaUeijckjyY2WjxWY/mU9v75v02jqpPlgzZZhELKv/3 ScE13HgxzPHAiDNXHJuQsJL8HxRbtl29aPV+406kQbolzfMudxXPU2hSIi8MDiYn hTJSZwp47bQngD9Ym8v+EdeExvRg2xLhlIuJc5j+34E9J5R9p/QC7Ru6YyzpESO5 olTzG/5Dt4V75q7mRkMtNiIWku9Ur5dtD+wjLAcQPmcoUuN0pX+rl2L4a7Wp+mqO rCuEZK5Y1S6/DBlu7UcBALe/T0OG8nzld4xLKZJR/oluuQSRXlRw6div4DaoqQMf 4PjjoG0+Hj2KS2aQm/JQ =oqiw -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Please help diagnosing a random production Tomcat 7.0.53 Internal Server Error!
2014-04-15 22:52 GMT+04:00 Ian Long ian.l...@opterus.com: Hi All, I need some help from all the tomcat experts out there! I am using tomcat behind apache httpd using mod_jk (1.2.39). About 50-100 times per day (out of many requests), I’m getting an internal server error from Tomcat (error 500), without any exceptions in my code, nor in Tomcat logs. I am only seeing the error in my New Relic application monitoring tool, and I can see them in the mod_jk logs if I turn on debug. Can you update to 1.2.40 released today? It fixes several issues. Is error 500 mentioned in Access log at Tomcat side? If an error happens at some early state of processing (in Connector, in CoyoteAdapter), then there may be nothing in the catalina/localhost/web application logs, unless you turn on debug logging at Tomcat side. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Please help diagnosing a random production Tomcat 7.0.53 Internal Server Error!
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Ian, On 4/15/14, 3:33 PM, Ian Long wrote: Thanks for the reply. It looks to me like tomcat just gave up partway through generating the request, I’m trying to figure out why. There are no exceptions in either my application logs or the tomcat log itself, which is frustrating. Definitely. You checked catalina.out (or wherever stdout goes) as well as your application's logs? Thanks, I’ll look into the executor. Apache matches what is set in my connector: IfModule prefork.c StartServers 8 MinSpareServers5 MaxSpareServers 20 ServerLimit 800 MaxClients 800 MaxRequestsPerChild 0 /IfModule Yes, the connector settings should be fine, there are usually less than 20 httpds. You mean 20 httpd prefork processes, right? That should be fine: it means you will need 20 connections available in Tomcat. Forgot to mention that it looks like tomcat returned around 50% of what the page should have been, before it hit the Internal Server Error. Have you run out of memory or anything like that? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTTY7wAAoJEBzwKT+lPKRYj6MQALTNWcCMZ7KI3+9PqBml7cId isbPrfJuSfVbta6lJXI8yuwjm6V/CvFc1l7WK2/qBosXO0jMopCZvCkJOOwVyAYt 6cozaLsH1YeFFfoOT6t7d/QhAjiWtlT+/sxX80dW/7t8uwbTQ7Bji01I3dtvYQsF f//HWfwDPSaxWBeXqZZ9bAG2uW7kiEExThlgQYbfcUnMPNB9Rc382GbI2/vIAtaR 9fWARiaLWfv4oaLzv67zAnFO/LV61HtLzA9PSy68do3AzZs0GvzKPPHlMdkobeGi lBUeSA8t9ZH7qetBaUUEto50cE5KnPtRVQG4bpA+9BrUyKHFxeyrB+rT3s1EhUlZ dH+QfioMEVQEAX/9tidA8pyWHiSNGYKCc2mAiIO2ahGWnx+IpUXOJz6bi0QnDJhp KeGrMrrV0R6fcUXoDiQzQGRTtWriJvl8VkP/eow3BpUeLO0ICdfYd9jOn5e0xtMV kO6X4N8aALyoTXtFm/0xTl01vXa5ZCWDdHRdtifcO9qAzHuGFYEjMaMeyUg08RAc BeSW3K8B2gAoXcilgOPAxuae9NRRwyius+tC0lLi/LvQRRbpAxBTV9Gv/BT/fbjU xndD+hVRiGcEoCmydngpmkGwqrroCfDWSyw4kYSxP9sGPRhNi3yPL3VlFBJXGUaC mfJtAJ7Rp6Ch6KKzY/oS =ag/e -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance - Java Profiler, JVM instrmentation
Hi Chris, On Tue, Apr 15, 2014 at 1:58 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 You need to use a Profiler for that. There are a number of fine Profilers available for Java. I use YourKit because they give free licenses to ASF committers. I'll look into a Java Profiler. Tomcat provides no such tools. You can use jconsole for a bit of profiling, but I've never used it heavily. I see. Honestly, if there is no budget for such tools, then there is no budget for improving the performance of your application. You'll have to convince someone with a higher level of clearance that customer needs are worth spending money. I am trying. I find Chris' example on writing filters to map to URL patterns for response-time metrics relevant. I would also like stall counts, concurrent invocations etc. What is a stall-count? How would you record concurrent invocations, etc.? So here is my understanding of these metrics: So if a request for a servlet or JSP exceeds a given time interval, that would be a stall. The interval may depend upon the application. In some cases, 10 seconds would be considered a stall, some cases, 30 seconds would be a stall. Similarly, how many times a servlet is invoked in a given time period would count as concurrent invocations. Intervals used for the reckoning here may be shorter - like 5 seconds - to make it more meaningful for concurrency values. Hi Leon, On Tue, Apr 15, 2014 at 3:45 PM, Leon Rosenberg rosenberg.l...@gmail.comwrote: last time I tried to use a profile on a production site it killed it within a second. Usually the performance overhead of a profiler is so huge, that you have no chance to run it in production. And real problems do not occur in test labs ;-) Leon I need to revisit New Relic. Many setups use these tools against QA and don't pass new code until load tests exercising the new code do not trip any thresholds. Which is a good workflow, although testing is controlled. I've only used them against production. Thanks, -Shanti
Re: Performance - Java Profiler, JVM instrmentation
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Leon, On 4/15/14, 3:45 PM, Leon Rosenberg wrote: On Tue, Apr 15, 2014 at 7:58 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Shanti, On 4/15/14, 10:51 AM, Shanti Suresh wrote: Taking it one step further, could I request some recommendations on how we might profile Java code running inside Tomcat? Often, I am stuck with finding out why an application is slow. It could be a consistent, progressive or a sudden problem. These applications do not expose metrics via MBeans. Say, for e.g., a vendor application which has been heavily customized in-house. You need to use a Profiler for that. There are a number of fine Profilers available for Java. I use YourKit because they give free licenses to ASF committers. Hello Chris, et al, last time I tried to use a profile on a production site it killed it within a second. +1 Profiling in production is a terrible idea. Usually the performance overhead of a profiler is so huge, that you have no chance to run it in production. And real problems do not occur in test labs ;-) Performance problems can usually be reproduced in labs. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTTY8pAAoJEBzwKT+lPKRYezUQALIQ8Vy124z/ZZZum3IqpEhy F5nk5rdDZLBtQF7wT8WH68fo254MNSvCPqfOLKfwDKGuMEl+RTxFg58k96Ii7e+S xC6wQwInJefdEtB1jz2E4hic8artLpAfjDPlXCT8DRHdLEWUF3qQ43H994/I3Ybs go41WFtP2+8aZaZnGQI664TGaERlrOJRu3AfMjU2VErG3kCJeeR/BFXhj7tZu/1Y 8cXt2CouFmKFYl5KNisI/3gfJ8X3hEGowZkGyswzSh+DkWIgxQMM7grnntHoxFtP iHlzQPdOB+gXmt2e2SISDc7HZNITRiBRvas8Q/+/xMiNJTbrRuKdRST6YKp3+AIt KuobyjJixJ0GuwjqW7P5/v+rP/PgF+eAT79FKtvKELUuRophh6e1ifO7osNkAkyp qCi+HoiThS9DfjJHdQnK9+uGMxRCirTvVIoNqHrJP4WJkxFeVRJQN1OKWaw+n/Xn SPAehFUZiUgS07NaR79lPvkHl8kXOAvrEEREwW6copv5c4BTQ/WlNYPG+2oRWmeZ YqTBS4bbZclV69Pg6cSSbfZGF/kmYq5zBKDOGIUHPF2zX4IoPmraU8IOLElIiS7o 0a5P64SLAsvOs4KnRJ4nFZSN80MwV6rODI2376+LlcdLjlQCmxoRiyNJMTOEqMUy dDn0cVrr7PzF0fpWoQN9 =F8p+ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance - Java Profiler, JVM instrmentation
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Shanti, On 4/15/14, 3:56 PM, Shanti Suresh wrote: So here is my understanding of these metrics: So if a request for a servlet or JSP exceeds a given time interval, that would be a stall. The interval may depend upon the application. In some cases, 10 seconds would be considered a stall, some cases, 30 seconds would be a stall. Can you afford to wait for the request to complete (late), or do you need to know immediately when a request is taking too long? In either case, this type of thing can be done easily with a filter. Whether you use JMX or not to report the condition is up to you. Similarly, how many times a servlet is invoked in a given time period would count as concurrent invocations. Intervals used for the reckoning here may be shorter - like 5 seconds - to make it more meaningful for concurrency values. Again, Filters are your friends. Feel free to publish the information via JMX as well. My presentation contains all the information you need for the JMX stuff. Everything else is pure Java. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTTY+rAAoJEBzwKT+lPKRYApcP/ROvCHanDW2RgYEneUPoG2u9 g/u2IKq8Noe8RmeQpVuU+k5i3tUo4hEigYsnmBiYHVq+CxY31hkMiPtuwMgHIrPd rHGfCy3Mgpsc3yLI2yuvYqCSs3pqR3+PYO1CiaxmgT52W8JWGoQjOOOdlk0BmseZ kF/vVtkT3qsM+c5YvghsqCnq9dyZK8YQXfzEigFdmLDzA42jAJ9gPhmoU4ZtfIXZ MKsn2xKgsz4XWaLsgLL2U/JX9Q77Sn/vXCcWktvP6wPZJDGFwf2jKKTuOF182NcT fevPjIysxi/R+zDUh+Dw7NkKty6HMx0x8DZHhKsy61YeK5fU+PgY8eyinGz5uTa2 +p2IgX2fUG5MiGe5oSO4g2oqLwNCqFsRbsvY5N3N4SnxuXaeQIsaIQKqYzgxBQyE d6BkPCGsObnpS59pqiQ2rF1cYNyu4z6pILGfS6ctZn43SSJ60HmhBZvE/qRtsezQ ycC2Wu8YzF8kJwCJyJJyPpg59m+hhlWYQ/Uk1BgTEhYpAYoAeOgrog+gdFXYt1RZ buI/t5vX6i38RjSe/L1zgDpx/x1x5kn4H/zhUfKM//4HaInfN2KG4cVejAhGtr5X kunZCIwh6rVlDIV167aIL6qDXinCpIIWJN2FhKO7khtCQrrbIIp7NYwWG0XfHYTn X4Fq69OJZRfKf7lDcSHv =uclh -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat logging with Log4j
Hello Christopher, What steps did you actually take? Steps on this site: http://mrhaki.blogspot.com/2011/02/configure-log4j-on-tomcat.html Downloaded new jars from extras for tomcat. tomcat-juli.jar and tomcat-juli-adapters.jar. Placed the tomcat-juli.jar file in our $CATALINA_BASE/bin directory. The file tomcat-juli-adapters.jar is copied to our $CATALINA_BASE/lib directory. Downloaded the latest log4j 1.2 library from the download page and copied to the $CATALIN_BASE/lib directory. Added the Log4j configuration file as it is on the link above. Disable the old Tomcat JUL logging configuration by deleting logging.properties Placed the Log4j configuration file. In the $CATALINA_BASE/lib directory I am not a java person but I believe we are using logback is what our java developers say, we do not state what files to log to but what to log and I think Tomcat logged it to the log file. statusListener class=ch.qos.logback.core.status.OnConsoleStatusListener/ appender name=STDOUT class=ch.qos.logback.core.ConsoleAppender encoder class=ch.qos.logback.core.encoder.LayoutWrappingEncoder layout class=com.donlen.common.utility.logging.DefaultAlignedLayout/ /encoder /appender Thanks! -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, April 15, 2014 2:22 PM To: Tomcat Users List Subject: Re: Tomcat logging with Log4j -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Scott, On 4/15/14, 2:50 PM, Scott Bailey wrote: We need to add log rotation and log size management to tomcat 7. Tried converting to Log4j steps from tomcat website (http://tomcat.apache.org/tomcat-7.0-doc/logging.html) but did not work, was able to get it to work from (http://mrhaki.blogspot.com/2011/02/configure-log4j-on-tomcat.html). It seems we are not getting any logging from our webapp though, and prior to this change it was getting logged in stdout and stderr. What steps did you actually take? Does something need to be changed in log4j.properties to still capture stdout and stderr to log file with Log4j? How are you logging from within your application? ServletContext.log()? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTTYbNAAoJEBzwKT+lPKRYHB4P/RUlDHyT2wQTt41S0YB/VbCR leQxOtDJYeO3WZ3qERNP1yzmX3GW2xKfxy8+5yXAgmPugO4LXbW2sj7a4VEDuR3D l5a/AkjqvsEdJFvWAB/77NFzWvcbOfiBT5Iw8AdkoprzaitdsD7UelLA7OlLbSCr EnLp1ZQmVJHEdAaFc0Rr2tH7SY1oSFZM19wmihBPTFZsOfnssiEnDnO6zhxgl9kG IlvNNuadefd0TxUiaNsncNYQatGjNsWHsjf6miYcEuJ2ZEz8x0YBzZv60MP0qiVi U/YvkvQCwGJU9wYvK9SjKpmJrunnB2dt3zKL724+qCw4D8h7qcccq6yytBXGRDha x4847oIbkvG7fqtljQTjfefuh8fFKalEVR8LP3huQtHqkpT4YRnRhJifFwZQmz1E aj7h/dz91F5pWmN673Fs8aO5LAM8qvsk7sT89QtBle0REeFZFEGPqGwBsDDV4sgE Shnr4JuR+xVyGCInHzV/zMDJ4EtVZVHRNFUnG9zS9Q5+FWfiOVc4cqxb4s6NgRKg mVhC8qharbzt/3I0nLNtMIxbz8c7hB2zNGBgjaQHEZ8BNp0+p3MtEOz9fGywi0pR CEL+HtjWbfaV+O8d3HSLyQMqtT5/ZOm10ZBG0s1yFjhaoxwz3yUjLLu2G3TeGJeD rRzHvGKrkAwWRehs+Zgm =g0p8 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org DISCLAIMER: This electronic mail message and any attached files contain information intended for the exclusive use of the intended addressee and may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify sender if you are an unintended recipient and delete the original message without making copies. Thank you.
Tomcat @ Windows Cyrillic
Hi all, I need to use cyrillic in my webapp. What I did: * Added -Dfile.encoding=utf-8 to the Java options * Added URIEncoding=UTF-8 to the connector in server.xml What I achieved: * When I send a request with cyrillic chars to the Tomcat, it's properly received. * When I send response with cyrillic chars out of the Tomcat, it's properly sent. What I did not achieved (and asking you kindly for help): * When my webapp, sitting in the Tomcat, sends a request containing cyrillic chars, these don't go out. So when I try to send out e.g. PrahaСимферополь, just Praha goes out. By sending out I mean calling cxf web service. When I call that webservice directly (from jar, not from Tomcat), everything works fine. Any idea? Many thanks, Petr P.S. Tomcat 7.0.53, Windows Server 2012 R2. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance - Java Profiler, JVM instrmentation
On Tue, Apr 15, 2014 at 3:59 PM, Christopher Schultz ch...@christopherschultz.net wrote: Can you afford to wait for the request to complete (late), or do you need to know immediately when a request is taking too long? My reasoning for measuring the long-running requests at a given period of time is so that you can co-relate it to other metrics at that given time. In my homegrown dashboard, I would graph the values returned alongside CPU, thread-use, Heap-size, connection-pool size etc. And then try to locate a signature of the problem. So just requests that have been inflight for the last n seconds at a given period of time would help. Or again, late requests, like you mention might work too. So at a given time, which requests completed late. Some of the tools out there have this troubleshooting down to a science and it makes life so easy for all parties! In either case, this type of thing can be done easily with a filter. Whether you use JMX or not to report the condition is up to you. Similarly, how many times a servlet is invoked in a given time period would count as concurrent invocations. Intervals used for the reckoning here may be shorter - like 5 seconds - to make it more meaningful for concurrency values. Again, Filters are your friends. Feel free to publish the information via JMX as well. My presentation contains all the information you need for the JMX stuff. Everything else is pure Java. Yes, I plan to explore that more. Thanks again, -Shanti
Re: Please help diagnosing a random production Tomcat 7.0.53 Internal Server Error!
Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Ian, On 4/15/14, 3:33 PM, Ian Long wrote: Thanks for the reply. It looks to me like tomcat just gave up partway through generating the request, I’m trying to figure out why. There are no exceptions in either my application logs or the tomcat log itself, which is frustrating. Definitely. You checked catalina.out (or wherever stdout goes) as well as your application's logs? Thanks, I’ll look into the executor. Apache matches what is set in my connector: IfModule prefork.c StartServers 8 MinSpareServers5 MaxSpareServers 20 ServerLimit 800 MaxClients 800 MaxRequestsPerChild 0 /IfModule Yes, the connector settings should be fine, there are usually less than 20 httpds. You mean 20 httpd prefork processes, right? That should be fine: it means you will need 20 connections available in Tomcat. Forgot to mention that it looks like tomcat returned around 50% of what the page should have been, before it hit the Internal Server Error. Have you run out of memory or anything like that? I was going to ask the same thing, slightly differently. I can think of a scenario which might result in the same kind of symptoms, only I am not sure if it makes sense, Java-wise. A request is recived by httpd, which passes it to Tomcat via mod_jk. Tomcat allocates a thread to handle the request, and this thread starts running the corresponding application (webapp). The webapp starts processing the request, produces some output, and then for some reason to be determined, it suddenly runs out of memory, and the thread running the application dies. Because Tomcat has temporarily run out of memory, there is no way for the application to write anything to the logs, because this would require allocating some additional memory to do so, and there isn't any available. So Tomcat just catches (a posteriori) the fact that the thread died, returning an error 500 to mod_jk and httpd. As soon as the offending thread dies, some memory is freed, and Tomcat appears to work normally again, including other requests to that same application, because those other requests do not cause the same spike in memory usage. Tomcat/Java experts : Could something like this happen, and would it match the symptoms as described by Ian ? And Ian, could it be that some requests to that application, because maybe of a parameter that is different from the other cases, could cause such a spike in memory requirements ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Please help diagnosing a random production Tomcat 7.0.53 Internal Server Error!
I don’t think it’s memory related - Tomcat is allocated an 8GB heap and according to New Relic it has never used more than 6.5G; there is also lots of PermGen space available. Cheers, Ian On April 15, 2014 at 4:18:11 PM, André Warnier (a...@ice-sa.com) wrote: Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Ian, On 4/15/14, 3:33 PM, Ian Long wrote: Thanks for the reply. It looks to me like tomcat just gave up partway through generating the request, I’m trying to figure out why. There are no exceptions in either my application logs or the tomcat log itself, which is frustrating. Definitely. You checked catalina.out (or wherever stdout goes) as well as your application's logs? Thanks, I’ll look into the executor. Apache matches what is set in my connector: IfModule prefork.c StartServers 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 800 MaxClients 800 MaxRequestsPerChild 0 /IfModule Yes, the connector settings should be fine, there are usually less than 20 httpds. You mean 20 httpd prefork processes, right? That should be fine: it means you will need 20 connections available in Tomcat. Forgot to mention that it looks like tomcat returned around 50% of what the page should have been, before it hit the Internal Server Error. Have you run out of memory or anything like that? I was going to ask the same thing, slightly differently. I can think of a scenario which might result in the same kind of symptoms, only I am not sure if it makes sense, Java-wise. A request is recived by httpd, which passes it to Tomcat via mod_jk. Tomcat allocates a thread to handle the request, and this thread starts running the corresponding application (webapp). The webapp starts processing the request, produces some output, and then for some reason to be determined, it suddenly runs out of memory, and the thread running the application dies. Because Tomcat has temporarily run out of memory, there is no way for the application to write anything to the logs, because this would require allocating some additional memory to do so, and there isn't any available. So Tomcat just catches (a posteriori) the fact that the thread died, returning an error 500 to mod_jk and httpd. As soon as the offending thread dies, some memory is freed, and Tomcat appears to work normally again, including other requests to that same application, because those other requests do not cause the same spike in memory usage. Tomcat/Java experts : Could something like this happen, and would it match the symptoms as described by Ian ? And Ian, could it be that some requests to that application, because maybe of a parameter that is different from the other cases, could cause such a spike in memory requirements ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Please help diagnosing a random production Tomcat 7.0.53 Internal Server Error!
Yes, I checked both the tomcat log (I’ve configured tomcat to use log4j) as well as my application logs. Yes, 20 httpd prefork processes. I don’t think it’s memory related, I have an 8GB heap and tomcat averages 5GB usage and peeks around 6.5 before garbage collection kicks in. Cheers, Ian On April 15, 2014 at 3:57:04 PM, Christopher Schultz (ch...@christopherschultz.net) wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Ian, On 4/15/14, 3:33 PM, Ian Long wrote: Thanks for the reply. It looks to me like tomcat just gave up partway through generating the request, I’m trying to figure out why. There are no exceptions in either my application logs or the tomcat log itself, which is frustrating. Definitely. You checked catalina.out (or wherever stdout goes) as well as your application's logs? Thanks, I’ll look into the executor. Apache matches what is set in my connector: IfModule prefork.c StartServers 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 800 MaxClients 800 MaxRequestsPerChild 0 /IfModule Yes, the connector settings should be fine, there are usually less than 20 httpds. You mean 20 httpd prefork processes, right? That should be fine: it means you will need 20 connections available in Tomcat. Forgot to mention that it looks like tomcat returned around 50% of what the page should have been, before it hit the Internal Server Error. Have you run out of memory or anything like that? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTTY7wAAoJEBzwKT+lPKRYj6MQALTNWcCMZ7KI3+9PqBml7cId isbPrfJuSfVbta6lJXI8yuwjm6V/CvFc1l7WK2/qBosXO0jMopCZvCkJOOwVyAYt 6cozaLsH1YeFFfoOT6t7d/QhAjiWtlT+/sxX80dW/7t8uwbTQ7Bji01I3dtvYQsF f//HWfwDPSaxWBeXqZZ9bAG2uW7kiEExThlgQYbfcUnMPNB9Rc382GbI2/vIAtaR 9fWARiaLWfv4oaLzv67zAnFO/LV61HtLzA9PSy68do3AzZs0GvzKPPHlMdkobeGi lBUeSA8t9ZH7qetBaUUEto50cE5KnPtRVQG4bpA+9BrUyKHFxeyrB+rT3s1EhUlZ dH+QfioMEVQEAX/9tidA8pyWHiSNGYKCc2mAiIO2ahGWnx+IpUXOJz6bi0QnDJhp KeGrMrrV0R6fcUXoDiQzQGRTtWriJvl8VkP/eow3BpUeLO0ICdfYd9jOn5e0xtMV kO6X4N8aALyoTXtFm/0xTl01vXa5ZCWDdHRdtifcO9qAzHuGFYEjMaMeyUg08RAc BeSW3K8B2gAoXcilgOPAxuae9NRRwyius+tC0lLi/LvQRRbpAxBTV9Gv/BT/fbjU xndD+hVRiGcEoCmydngpmkGwqrroCfDWSyw4kYSxP9sGPRhNi3yPL3VlFBJXGUaC mfJtAJ7Rp6Ch6KKzY/oS =ag/e -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat @ Windows Cyrillic
Petr Nemecek wrote: Hi all, I need to use cyrillic in my webapp. What I did: * Added -Dfile.encoding=utf-8 to the Java options * Added URIEncoding=UTF-8 to the connector in server.xml What I achieved: * When I send a request with cyrillic chars to the Tomcat, it's properly received. * When I send response with cyrillic chars out of the Tomcat, it's properly sent. What I did not achieved (and asking you kindly for help): * When my webapp, sitting in the Tomcat, sends a request containing cyrillic chars, these don't go out. So when I try to send out e.g. PrahaСимферополь, just Praha goes out. By sending out I mean calling cxf web service. When I call that webservice directly (from jar, not from Tomcat), everything works fine. Any idea? When an application (running under tomcat or not) sends a request to some external service, it uses some code to do that, which is not part of the tomcat code. Which code/library are you using for that ? In how much could this code/library be influenced by the fact that it is running under the JVM which runs Tomcat (and the settings of that Tomcat JVM) ? More explicitly : a Java application doesn't just run. It is being run by the JVM that runs it. That JVM runs with certain settings (say, system properties) which influence the way in which it runs the applications which it runs. On thing which has happened to me in the past (and I really don't know if this is still applicable) is as follows : Tomcat starts, under some JVM settings appropriate for your application. Then it runs various applications, including yours. One of the other applications, for whatever reason of its own, changes a JVM system property. And because this is a JVM-global property, this new setting now interferes with your application when it runs. (And the case of which I am talking, happened to concern some default language setting; and this caused my application to suddenly start outputting messages in English instead of the expected German). This would of course not happen, when you run your application stand-alone, because then there are no other applications to mess with the JVM system properties. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Please help diagnosing a random production Tomcat 7.0.53 Internal Server Error!
Ian, On this list, it is kind of frowned-upon to top post. It is preferred when people answer a question, below the question. Keeps things more logical in the reading sequence, and avoids having to scroll down to guess what you are responding to. Ian Long wrote: Yes, I checked both the tomcat log (I’ve configured tomcat to use log4j) as well as my application logs. Yes, 20 httpd prefork processes. I don’t think it’s memory related, I have an 8GB heap and tomcat averages 5GB usage and peeks around 6.5 before garbage collection kicks in. Of course we do not know (yet) either what the cause of your problem is. But we know that Tomcat would normally write something in its logs, when a server error 500 happens. So, - either Tomcat and /or your application wrote something to a logfile, and you have not yet found that logfile - or else Tomcat and/or your application crashed, but did not write anything to the logs. In that last case, one of the most likely causes for such a behaviour is running out of memory. Whether you believe that this is possible or not is your opinion. But it is of the nature of software bugs, to be unexpected. If they were expected, they would have been corrected already. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Please help diagnosing a random production Tomcat 7.0.53 Internal Server Error!
On April 15, 2014 at 4:58:28 PM, André Warnier (a...@ice-sa.com(mailto:a...@ice-sa.com)) wrote: Ian, On this list, it is kind of frowned-upon to top post. It is preferred when people answer a question, below the question. Keeps things more logical in the reading sequence, and avoids having to scroll down to guess what you are responding to. Ian Long wrote: Yes, I checked both the tomcat log (I’ve configured tomcat to use log4j) as well as my application logs. Yes, 20 httpd prefork processes. I don’t think it’s memory related, I have an 8GB heap and tomcat averages 5GB usage and peeks around 6.5 before garbage collection kicks in. Of course we do not know (yet) either what the cause of your problem is. But we know that Tomcat would normally write something in its logs, when a server error 500 happens. So, - either Tomcat and /or your application wrote something to a logfile, and you have not yet found that logfile - or else Tomcat and/or your application crashed, but did not write anything to the logs. In that last case, one of the most likely causes for such a behaviour is running out of memory. Whether you believe that this is possible or not is your opinion. But it is of the nature of software bugs, to be unexpected. If they were expected, they would have been corrected already. Ok, thanks, didn’t know about the top post issue. I have tomcat configured to log via log4j, and then there is my application log, those are the only two logs, and neither contains anything. It’s not about believing, I have monitoring software that gives me precise information about memory use and there is no indication of a problem there. Thanks, Ian - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Please help diagnosing a random production Tomcat 7.0.53 Internal Server Error!
Ian Long wrote: On April 15, 2014 at 4:58:28 PM, André Warnier (a...@ice-sa.com(mailto:a...@ice-sa.com)) wrote: Ian, On this list, it is kind of frowned-upon to top post. It is preferred when people answer a question, below the question. Keeps things more logical in the reading sequence, and avoids having to scroll down to guess what you are responding to. Ian Long wrote: Yes, I checked both the tomcat log (I’ve configured tomcat to use log4j) as well as my application logs. Yes, 20 httpd prefork processes. I don’t think it’s memory related, I have an 8GB heap and tomcat averages 5GB usage and peeks around 6.5 before garbage collection kicks in. Of course we do not know (yet) either what the cause of your problem is. But we know that Tomcat would normally write something in its logs, when a server error 500 happens. So, - either Tomcat and /or your application wrote something to a logfile, and you have not yet found that logfile - or else Tomcat and/or your application crashed, but did not write anything to the logs. In that last case, one of the most likely causes for such a behaviour is running out of memory. Whether you believe that this is possible or not is your opinion. But it is of the nature of software bugs, to be unexpected. If they were expected, they would have been corrected already. Ok, thanks, didn’t know about the top post issue. I have tomcat configured to log via log4j, and then there is my application log, those are the only two logs, and neither contains anything. It’s not about believing, I have monitoring software that gives me precise information about memory use and there is no indication of a problem there. Would that monitoring software detect a very short occasional spike in the usage of memory, just before the thread running that application is blown out of the water and the memory usage returns to normal ? Or is it something that updates its data on a 5-second interval and it just always misses the significant event ? Honestly, I am just fishing and trying to find a clue (or rather, trying to help you find a clue). But some problems are just like that. You can only carefully eliminate the possible causes one after the other until you're left with one that you cannot eliminate. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Which tcnative to replace for Heartbleed?
I deploy Tomcat 7 in both 64 and 32 bit environments. When I deploy/upgrade, I download Tomcat from this page: http://tomcat.apache.org/download-70.cgi, downloading both the 32-bit Windows and 64-bit Windows zip files. I would like to make sure that my Tomcat deployments are secure from the OpenSSL Heartbleed bug, and my understanding is that I simply need to replace tcnative-1.dll in my download with one from this page: http://apache.org/dist/tomcat/tomcat-connectors/native/1.1.30/binaries/. But which one? I assume I don't need OCSP-do I? But then in the download there are 3 different versions, one at the top level, one in i64 and one in x64. Can I assume that the top level one is 32 bit and the x64 one is 64 bit? Of course, it would be useful if there were simply a new release of Tomcat, or a readily available guide for current users on how to protect ourselves from this issue. Knowing whether an updated Heartbleed-free version of Windows Tomcat was coming in the next few days would resolve this issue as well. Thanks, Scott
Re: Which tcnative to replace for Heartbleed?
2014-04-16 0:46 GMT+04:00 Scott Johnson sjohn...@dag.com: I deploy Tomcat 7 in both 64 and 32 bit environments. When I deploy/upgrade, I download Tomcat from this page: http://tomcat.apache.org/download-70.cgi, downloading both the 32-bit Windows and 64-bit Windows zip files. I would like to make sure that my Tomcat deployments are secure from the OpenSSL Heartbleed bug, and my understanding is that I simply need to replace tcnative-1.dll in my download with one from this page: http://apache.org/dist/tomcat/tomcat-connectors/native/1.1.30/binaries/. Where did you get that link? A policy is that we do not advertise direct links to the ASF server, but suggest using the mirrors. http://tomcat.apache.org/download-native.cgi - You may download them from HERE (a link) Though the ASF server contains the MD% and ASC files. (Those are not mirrored). But which one? I assume I don't need OCSP-do I? Yes, that is correct. But then in the download there are 3 different versions, one at the top level, one in i64 and one in x64. Can I assume that the top level one is 32 bit and the x64 one is 64 bit? Yes, that is correct. Of course, it would be useful if there were simply a new release of Tomcat, or a readily available guide for current users on how to protect ourselves from this issue. Knowing whether an updated Heartbleed-free version of Windows Tomcat was coming in the next few days would resolve this issue as well. A work is going, but that will take some time. There are still bugs that need fixing before cutting a release. The release vote itself will take 3 days (72h). A guide is on the wiki, http://wiki.apache.org/tomcat/Security/Heartbleed Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Which tcnative to replace for Heartbleed?
Thanks for your reply, that clears up just about everything. I got the link directly from the Bugzilla bug where this issue was reported, by the way. Scott -Original Message- From: Konstantin Kolinko [mailto:knst.koli...@gmail.com] Sent: Tuesday, April 15, 2014 3:03 PM To: Tomcat Users List Subject: Re: Which tcnative to replace for Heartbleed? 2014-04-16 0:46 GMT+04:00 Scott Johnson sjohn...@dag.com: I deploy Tomcat 7 in both 64 and 32 bit environments. When I deploy/upgrade, I download Tomcat from this page: http://tomcat.apache.org/download-70.cgi, downloading both the 32-bit Windows and 64-bit Windows zip files. I would like to make sure that my Tomcat deployments are secure from the OpenSSL Heartbleed bug, and my understanding is that I simply need to replace tcnative-1.dll in my download with one from this page: http://apache.org/dist/tomcat/tomcat-connectors/native/1.1.30/binaries/. Where did you get that link? A policy is that we do not advertise direct links to the ASF server, but suggest using the mirrors. http://tomcat.apache.org/download-native.cgi - You may download them from HERE (a link) Though the ASF server contains the MD% and ASC files. (Those are not mirrored). But which one? I assume I don't need OCSP-do I? Yes, that is correct. But then in the download there are 3 different versions, one at the top level, one in i64 and one in x64. Can I assume that the top level one is 32 bit and the x64 one is 64 bit? Yes, that is correct. Of course, it would be useful if there were simply a new release of Tomcat, or a readily available guide for current users on how to protect ourselves from this issue. Knowing whether an updated Heartbleed-free version of Windows Tomcat was coming in the next few days would resolve this issue as well. A work is going, but that will take some time. There are still bugs that need fixing before cutting a release. The release vote itself will take 3 days (72h). A guide is on the wiki, http://wiki.apache.org/tomcat/Security/Heartbleed Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Please help diagnosing a random production Tomcat 7.0.53 Internal Server Error!
On Tue, 2014-04-15 at 17:12 -0400, Ian Long wrote: Ian Long wrote: I have tomcat configured to log via log4j, and then there is my application log, those are the only two logs, and neither contains anything. They're empty? Are you sure the logs are writable? How much free space is available on the file system where the logs reside? It’s not about believing, I have monitoring software that gives me precise information about memory use and there is no indication of a problem there. Thanks, Ian - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Please help diagnosing a random production Tomcat 7.0.53 Internal Server Error!
On April 15, 2014 at 6:50:05 PM, Tim Watts (t...@cliftonfarm.org(mailto:t...@cliftonfarm.org)) wrote: On Tue, 2014-04-15 at 17:12 -0400, Ian Long wrote: Ian Long wrote: I have tomcat configured to log via log4j, and then there is my application log, those are the only two logs, and neither contains anything. They're empty? Are you sure the logs are writable? How much free space is available on the file system where the logs reside? It’s not about believing, I have monitoring software that gives me precise information about memory use and there is no indication of a problem there. Thanks, Ian - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Sorry, I should have been more clear. No, they are not empty, things are being logged in both files, just not specifically for this problem. There are no errors in the logs corresponding to the time I see the error recorded in New Relic. There is more than 100GB of free space on the drive. Cheers, Ian - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
tomcat 7.0.53 error
Hi, I want migrate from tomcat 7.0.52 to 7.0.53. My tomcat is launched in standalone mode from command line (linux level3)Currently i have a servlet that create image to disk and only with last version i have following error messageException in thread http-bio-8080-exec-10 java.lang.InternalError: Can't connect to X11 window server using ':0.0' as the value of the DISPLAY variable.Same servlet running perfect with previous version. RegardsPhil
