Re: Performance - Java Profiler, JVM instrmentation
Hi Shanti,
On 04/15/2014 09:56 PM, Shanti Suresh wrote:
[...]
I find Chris' example on writing filters to map to URL patterns for
response-time metrics relevant. I would also like stall counts,
concurrent invocations etc.
What is a stall-count? How would you record concurrent invocations,
etc.?
So here is my understanding of these metrics:
So if a request for a servlet or JSP exceeds a given time interval, that
would be a stall. The interval may depend upon the application. In some
cases, 10 seconds would be considered a stall, some cases, 30 seconds would
be a stall.
This can be done enabling the access log and adding a %D on the log
format string, here's
what i add to server.xml in tomcat 6:
!-- --
Valve className=org.apache.catalina.valves.AccessLogValve
directory=logs
prefix=access. suffix=.log resolveHosts=false
pattern='%h %u %t %r %s %b %I %D'
buffered=false /
then you get another log file, in this case access.DATE.log where the
last entry is the time in milliseconds
it took to complete the request.
Than just do a:
cat access.DATE.log | awk '{ if ($NF DURATION) { print $0 } }'
Hope you got the idea
Similarly, how many times a servlet is invoked in a given time period would
count as concurrent invocations. Intervals used for the reckoning here may
be shorter - like 5 seconds - to make it more meaningful for concurrency
values.
You can use the access log for this too
[..]
Frederik
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Performance - Java Profiler, JVM instrmentation
Greetings, Chris' presentation on monitoring Tomcat is really nice work. I found that quite useful. Taking it one step further, could I request some recommendations on how we might profile Java code running inside Tomcat? Often, I am stuck with finding out why an application is slow. It could be a consistent, progressive or a sudden problem. These applications do not expose metrics via MBeans. Say, for e.g., a vendor application which has been heavily customized in-house. Some metrics that I find useful during these times are things like concurrent invocations, stall counts on components, call-stack, response-rate etc. Java Melody has a nice built-in dashboard of metrics. Co-relating metrics like that is powerful and helps isolate relatively easy problems. I find that the metrics skim the surface of more involved problems. In Tomcat, is there a way to go deeper into the performance of the code for root-cause analysis and isolate a section of the code or a flow in the code for troubleshooting? How would one go about getting to that place? Let's say, there is no budget for purchasing tools in that space. I find Chris' example on writing filters to map to URL patterns for response-time metrics relevant. I would also like stall counts, concurrent invocations etc. Greatly appreciate your thoughts and opinions. Thanks, -Shanti
Re: Performance - Java Profiler, JVM instrmentation
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Shanti, On 4/15/14, 10:51 AM, Shanti Suresh wrote: Taking it one step further, could I request some recommendations on how we might profile Java code running inside Tomcat? Often, I am stuck with finding out why an application is slow. It could be a consistent, progressive or a sudden problem. These applications do not expose metrics via MBeans. Say, for e.g., a vendor application which has been heavily customized in-house. You need to use a Profiler for that. There are a number of fine Profilers available for Java. I use YourKit because they give free licenses to ASF committers. In Tomcat, is there a way to go deeper into the performance of the code for root-cause analysis and isolate a section of the code or a flow in the code for troubleshooting? How would one go about getting to that place? Let's say, there is no budget for purchasing tools in that space. Tomcat provides no such tools. You can use jconsole for a bit of profiling, but I've never used it heavily. Honestly, if there is no budget for such tools, then there is no budget for improving the performance of your application. You'll have to convince someone with a higher level of clearance that customer needs are worth spending money. I find Chris' example on writing filters to map to URL patterns for response-time metrics relevant. I would also like stall counts, concurrent invocations etc. What is a stall-count? How would you record concurrent invocations, etc.? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTTXM7AAoJEBzwKT+lPKRY/g8P/3hdbqYf5I2s0LaxiHexJ27I tTouN/Gk0+kmOapdoPUxXECC8q/LF51I7aCF5L2D/XY5Jgu5Vx5grwvoZWVdAS5N OB54gLvOq1toVSGTI7C7POqg8S7gqdj1Ec5BCrxiI0HGmSWNYcDQoFNWXL+srE6L xes6o1Kfhlvs/cjmhWagooaGvaNY1Wu7DyhJowaTmWzDecvEmIS+A3ysuSoB+g4B Myc61ZR4Imul6+0iRm59dWjE5ID9cqK9jr5NzFUIcy2zJaCIkNV1SKnliWkZBgp/ b/B7pUWfTnjUiX4lEDjH9WYL6cXO/fkwL70yue1iki/u+MqOJn7c+Qyo2VzRtn1W iYHpm5BKngNER8r/4AhzV7ZvqGlmqOoLHqmh88Seiq8AAcloLdDkXWhR/GQzHkuk +ESCRv5qUmmDwOdk1wcuMjGOk10+tN5zc2ZfrJlchYxVKvPaxHFcFVSczlcb+Bok heetKDK2PZlYlzQ0iACwBHLzmaU/Jycmv3+zVOucBmyeZZBUX6GpAjG2lsh0QPlG DhIPj1PHuC6OYsIGbVeLFirFRo/hS+XwACaY/qBVyyGd9k6NRjtpQ9RMQvb2dUGu CPh52SBzgyOGGu9dZ1EaYSC7Vl21DySDmWr8eUMxIjEYinr0rc5Mv1VqOQEWaWy9 rixRAxhweuazzCqGnat7 =0i+1 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance - Java Profiler, JVM instrmentation
On Tue, Apr 15, 2014 at 4:51 PM, Shanti Suresh sha...@umich.edu wrote: Greetings, Hello Shanti, Chris' presentation on monitoring Tomcat is really nice work. I found that quite useful. Taking it one step further, could I request some recommendations on how we might profile Java code running inside Tomcat? Often, I am stuck with finding out why an application is slow. It could be a consistent, progressive or a sudden problem. These applications do not expose metrics via MBeans. Say, for e.g., a vendor application which has been heavily customized in-house. Some metrics that I find useful during these times are things like concurrent invocations, stall counts on components, call-stack, response-rate etc. Java Melody has a nice built-in dashboard of metrics. Co-relating metrics like that is powerful and helps isolate relatively easy problems. I find that the metrics skim the surface of more involved problems. In Tomcat, is there a way to go deeper into the performance of the code for root-cause analysis and isolate a section of the code or a flow in the code for troubleshooting? How would one go about getting to that place? Let's say, there is no budget for purchasing tools in that space. I find Chris' example on writing filters to map to URL patterns for response-time metrics relevant. I would also like stall counts, concurrent invocations etc. There are tools that are doing exactly that for about 7 years out now. You can go to http://newrelic.com and get it for as much as 150 USD per server. Or you can get all the same for free from http://www.moskito.org. And more. regards Leon Greatly appreciate your thoughts and opinions. Thanks, -Shanti
Re: Performance - Java Profiler, JVM instrmentation
On Tue, Apr 15, 2014 at 7:58 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Shanti, On 4/15/14, 10:51 AM, Shanti Suresh wrote: Taking it one step further, could I request some recommendations on how we might profile Java code running inside Tomcat? Often, I am stuck with finding out why an application is slow. It could be a consistent, progressive or a sudden problem. These applications do not expose metrics via MBeans. Say, for e.g., a vendor application which has been heavily customized in-house. You need to use a Profiler for that. There are a number of fine Profilers available for Java. I use YourKit because they give free licenses to ASF committers. Hello Chris, et al, last time I tried to use a profile on a production site it killed it within a second. Usually the performance overhead of a profiler is so huge, that you have no chance to run it in production. And real problems do not occur in test labs ;-) Leon
Re: Performance - Java Profiler, JVM instrmentation
Hi Chris, On Tue, Apr 15, 2014 at 1:58 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 You need to use a Profiler for that. There are a number of fine Profilers available for Java. I use YourKit because they give free licenses to ASF committers. I'll look into a Java Profiler. Tomcat provides no such tools. You can use jconsole for a bit of profiling, but I've never used it heavily. I see. Honestly, if there is no budget for such tools, then there is no budget for improving the performance of your application. You'll have to convince someone with a higher level of clearance that customer needs are worth spending money. I am trying. I find Chris' example on writing filters to map to URL patterns for response-time metrics relevant. I would also like stall counts, concurrent invocations etc. What is a stall-count? How would you record concurrent invocations, etc.? So here is my understanding of these metrics: So if a request for a servlet or JSP exceeds a given time interval, that would be a stall. The interval may depend upon the application. In some cases, 10 seconds would be considered a stall, some cases, 30 seconds would be a stall. Similarly, how many times a servlet is invoked in a given time period would count as concurrent invocations. Intervals used for the reckoning here may be shorter - like 5 seconds - to make it more meaningful for concurrency values. Hi Leon, On Tue, Apr 15, 2014 at 3:45 PM, Leon Rosenberg rosenberg.l...@gmail.comwrote: last time I tried to use a profile on a production site it killed it within a second. Usually the performance overhead of a profiler is so huge, that you have no chance to run it in production. And real problems do not occur in test labs ;-) Leon I need to revisit New Relic. Many setups use these tools against QA and don't pass new code until load tests exercising the new code do not trip any thresholds. Which is a good workflow, although testing is controlled. I've only used them against production. Thanks, -Shanti
Re: Performance - Java Profiler, JVM instrmentation
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Leon, On 4/15/14, 3:45 PM, Leon Rosenberg wrote: On Tue, Apr 15, 2014 at 7:58 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Shanti, On 4/15/14, 10:51 AM, Shanti Suresh wrote: Taking it one step further, could I request some recommendations on how we might profile Java code running inside Tomcat? Often, I am stuck with finding out why an application is slow. It could be a consistent, progressive or a sudden problem. These applications do not expose metrics via MBeans. Say, for e.g., a vendor application which has been heavily customized in-house. You need to use a Profiler for that. There are a number of fine Profilers available for Java. I use YourKit because they give free licenses to ASF committers. Hello Chris, et al, last time I tried to use a profile on a production site it killed it within a second. +1 Profiling in production is a terrible idea. Usually the performance overhead of a profiler is so huge, that you have no chance to run it in production. And real problems do not occur in test labs ;-) Performance problems can usually be reproduced in labs. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTTY8pAAoJEBzwKT+lPKRYezUQALIQ8Vy124z/ZZZum3IqpEhy F5nk5rdDZLBtQF7wT8WH68fo254MNSvCPqfOLKfwDKGuMEl+RTxFg58k96Ii7e+S xC6wQwInJefdEtB1jz2E4hic8artLpAfjDPlXCT8DRHdLEWUF3qQ43H994/I3Ybs go41WFtP2+8aZaZnGQI664TGaERlrOJRu3AfMjU2VErG3kCJeeR/BFXhj7tZu/1Y 8cXt2CouFmKFYl5KNisI/3gfJ8X3hEGowZkGyswzSh+DkWIgxQMM7grnntHoxFtP iHlzQPdOB+gXmt2e2SISDc7HZNITRiBRvas8Q/+/xMiNJTbrRuKdRST6YKp3+AIt KuobyjJixJ0GuwjqW7P5/v+rP/PgF+eAT79FKtvKELUuRophh6e1ifO7osNkAkyp qCi+HoiThS9DfjJHdQnK9+uGMxRCirTvVIoNqHrJP4WJkxFeVRJQN1OKWaw+n/Xn SPAehFUZiUgS07NaR79lPvkHl8kXOAvrEEREwW6copv5c4BTQ/WlNYPG+2oRWmeZ YqTBS4bbZclV69Pg6cSSbfZGF/kmYq5zBKDOGIUHPF2zX4IoPmraU8IOLElIiS7o 0a5P64SLAsvOs4KnRJ4nFZSN80MwV6rODI2376+LlcdLjlQCmxoRiyNJMTOEqMUy dDn0cVrr7PzF0fpWoQN9 =F8p+ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance - Java Profiler, JVM instrmentation
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Shanti, On 4/15/14, 3:56 PM, Shanti Suresh wrote: So here is my understanding of these metrics: So if a request for a servlet or JSP exceeds a given time interval, that would be a stall. The interval may depend upon the application. In some cases, 10 seconds would be considered a stall, some cases, 30 seconds would be a stall. Can you afford to wait for the request to complete (late), or do you need to know immediately when a request is taking too long? In either case, this type of thing can be done easily with a filter. Whether you use JMX or not to report the condition is up to you. Similarly, how many times a servlet is invoked in a given time period would count as concurrent invocations. Intervals used for the reckoning here may be shorter - like 5 seconds - to make it more meaningful for concurrency values. Again, Filters are your friends. Feel free to publish the information via JMX as well. My presentation contains all the information you need for the JMX stuff. Everything else is pure Java. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTTY+rAAoJEBzwKT+lPKRYApcP/ROvCHanDW2RgYEneUPoG2u9 g/u2IKq8Noe8RmeQpVuU+k5i3tUo4hEigYsnmBiYHVq+CxY31hkMiPtuwMgHIrPd rHGfCy3Mgpsc3yLI2yuvYqCSs3pqR3+PYO1CiaxmgT52W8JWGoQjOOOdlk0BmseZ kF/vVtkT3qsM+c5YvghsqCnq9dyZK8YQXfzEigFdmLDzA42jAJ9gPhmoU4ZtfIXZ MKsn2xKgsz4XWaLsgLL2U/JX9Q77Sn/vXCcWktvP6wPZJDGFwf2jKKTuOF182NcT fevPjIysxi/R+zDUh+Dw7NkKty6HMx0x8DZHhKsy61YeK5fU+PgY8eyinGz5uTa2 +p2IgX2fUG5MiGe5oSO4g2oqLwNCqFsRbsvY5N3N4SnxuXaeQIsaIQKqYzgxBQyE d6BkPCGsObnpS59pqiQ2rF1cYNyu4z6pILGfS6ctZn43SSJ60HmhBZvE/qRtsezQ ycC2Wu8YzF8kJwCJyJJyPpg59m+hhlWYQ/Uk1BgTEhYpAYoAeOgrog+gdFXYt1RZ buI/t5vX6i38RjSe/L1zgDpx/x1x5kn4H/zhUfKM//4HaInfN2KG4cVejAhGtr5X kunZCIwh6rVlDIV167aIL6qDXinCpIIWJN2FhKO7khtCQrrbIIp7NYwWG0XfHYTn X4Fq69OJZRfKf7lDcSHv =uclh -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Performance - Java Profiler, JVM instrmentation
On Tue, Apr 15, 2014 at 3:59 PM, Christopher Schultz ch...@christopherschultz.net wrote: Can you afford to wait for the request to complete (late), or do you need to know immediately when a request is taking too long? My reasoning for measuring the long-running requests at a given period of time is so that you can co-relate it to other metrics at that given time. In my homegrown dashboard, I would graph the values returned alongside CPU, thread-use, Heap-size, connection-pool size etc. And then try to locate a signature of the problem. So just requests that have been inflight for the last n seconds at a given period of time would help. Or again, late requests, like you mention might work too. So at a given time, which requests completed late. Some of the tools out there have this troubleshooting down to a science and it makes life so easy for all parties! In either case, this type of thing can be done easily with a filter. Whether you use JMX or not to report the condition is up to you. Similarly, how many times a servlet is invoked in a given time period would count as concurrent invocations. Intervals used for the reckoning here may be shorter - like 5 seconds - to make it more meaningful for concurrency values. Again, Filters are your friends. Feel free to publish the information via JMX as well. My presentation contains all the information you need for the JMX stuff. Everything else is pure Java. Yes, I plan to explore that more. Thanks again, -Shanti
