Overriding Close method of a Output print writer through HttpServletResponseWrapper
Hi, I need a help for the following problem - I have a web application where cookies are getting added multiple times and because of that response header is becoming big containing many duplicate cookies. Now the web application is so big that it is nearly impossible to correct those cookie logic in a short term. So I have planned to use a HttpServletResponseWrapper in a Filter. It will contain a HashMap. I have overridden the addcookie method of the response wrapper. Now the cookies will be added to the map instead of response. So it will not have any duplicate cookie. Now I have created a Custom Servlet Output Stream and Print writer and overridden the close method. Before closing the stream/writer I will take the cookies from map and will add to the actual response header. Problem is it works fine with pure servlet. But the close method of the stream/writer not getting called when using with JSP. What can be the issue - I have overridden the getOutPutStream/Getwriter method and those method will return my custom classes. I have also overridden the flushbuffer method. Thanks, Arnab Ghosh
Re: Help in diagnosing server unresponsiveness
In the Java world, most people would only call it a consider it a cluster if the app servers actually know about each other -- for instance, if you are using session replication. IMO session replication is a dog, and there are better ways to achieve similar goals that yield much higher performance. I am curious to hear your thoughts - care to elaborate? Thanks Chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
On 01/02/2013 20:08, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Zoran, On 1/31/13 8:36 PM, Zoran Avtarovski wrote: We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. Can you describe the crash in more detail? OOME? IF so, what kind (heap or PermGen)? Lock-up (deadlock, etc)? Actual JVM crash (produces a core dump or native stack dump)? I would go in that direction too. Enable logs and core or stack dumps and analyze them. Be sure you are not restarting Tomcat in your crontab (i had a server which was restarted once a week and masked some memory starvation). In my case I can tell you I had to end up disabling JaveMelody (it was provoking some side effects in our webapp as not managing international chars in the right way). If reports from Javamelody are not giving you any clue, beware that Javamelody has its own memory overhead (not much in your case but in my case it was around 200 Mb of heap in a 1 Gb virtual server). I followed Chris directions, I got stack dumps after a server crash and analyzed it with eclipse analyzer. I realized our programmer decided to load too many objects in memory than the server could cope with. So no memory leak but bad memory management was the root cause. Regards, Miguel - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Different webapp paths on different hosts
On 1/28/2013 7:13 PM, Christopher Schultz wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
bxqdev,
On 1/26/13 1:09 AM, bxqdev wrote:
On 1/26/2013 1:05 AM, Christopher Schultz wrote: Dude,
On 1/25/13 12:38 AM, bxqdev wrote:
i want to have one instance of webapp,
No problem.
which has two paths,
Problem: one context = one path = one instance
If you want two separate paths, you need two separate contexts.
and each of the paths serves the root of different domains
That just doesn't make any sense. The root of a domain is, by
definition, / (contextPath=), so there aren't any other paths
to consider.
looks like the goal is not reachable with plain tomcat
configuration.
once again: * one webapp * has 2 different paths: /path1
and /path2 * served on 2 domains * when user goes to
http://path1.com/ - {webapp}/path1 is served as root path *
when user goes to http://path2.com/ - {webapp}/path2 is
served as root path
is it possible?
What you are saying is nonsensical (specifically, the part where
you say that you want /path1 to be mapped to root.
If you want two virtual hosts, then define two virtual hosts. If
you want an app on each host configured as ROOT, then make your
two virtual hosts have two separate appBase directories (which
should be a requirement in the first place of two virtual hosts)
and toss a copy of your WAR file in each appBase, both named
ROOT.war (cApITaLS matter).
You can't deploy a single copy (in memory) of your webapp and
expect to get all your requirements above met. If you want to have
a single WAR file on the disk and deploy it multiple times, you
*can* do that but I'm not sure why anyone would want to.
looks like the goal is not reachable with plain tomcat
configuration.
I seriously don't understand your goal or your problem. It sounds
like you want two separate webapps smashed into a single context where
one webapp gets served by one domain and another webapps gets served
by the other domain. I don't know why you don't just split them up
into two separate webapps just like $DIETY intended.
if that was the solution, i wouldn't ask the question in the first place.
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEAREIAAYFAlEGlY0ACgkQ9CaO5/Lv0PD86wCfcid6tNjOzzR0BJZh2tF5672y
09kAmwQ/o+S3jP3Ii033Wfm1UkVmecXs
=shqR
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Different webapp paths on different hosts
On 1/28/2013 7:11 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 bxqdev, On 1/26/13 3:04 PM, bxqdev wrote: On 1/26/2013 9:36 PM, Caldarale, Charles R wrote: From: cjder...@gmail.com [mailto:cjder...@gmail.com] On Behalf Of chris derham Subject: Re: Different webapp paths on different hosts To my mind if you deploy the app as ROOT.war, as long as DNS is configured correctly, that single context will serve responses to http://app1.com/ and http://app2.com/. This appears solves the explicit requirement stated above. Which is what I suggested back on 23 January: Assuming path1.com and path2.com evaluate to the same IP address (or at least to the same system), this will happen automatically with a single Host element in the Tomcat configuration. But the OP refuses to supply any information about why that's not sufficient. - Chuck in this case http://app1.com/ and http://app2.com/ will serve the same content. i need different content on different domains. Something actually made a little pop in my brain when I read that. I gotta go... I think I smell toast burning... i take it as you don't know the correct answer. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEGlRIACgkQ9CaO5/Lv0PBywQCeO1wxbRMQX6reeKVzCaYCjIY8 F74An104rdsUKFUucCNBZgfBC3Mh+7Lr =d8JI -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Different webapp paths on different hosts
To my mind if you deploy the app as ROOT.war, as long as DNS is configured correctly, that single context will serve responses to http://app1.com/ and http://app2.com/. This appears solves the explicit requirement stated above. But the OP refuses to supply any information about why that's not sufficient. in this case http://app1.com/ and http://app2.com/ will serve the same content. i need different content on different domains. You asked how can I serve content from one domain to one host, and a different set of content for another domain. You were told that the easiest way is to deploy two separate apps. You said that isn't what you want, but did not explain why. What technical reason is this not a suitable solution? If you really want a single app deployed that sends different content depending on the host used, then you will need to code that inside your app. This isn't really application/web server issue, its something that needs to be handled inside the web app Something actually made a little pop in my brain when I read that. I gotta go... I think I smell toast burning... i take it as you don't know the correct answer. You asked an incredibly badly question, and didn't provide a clear explanation of what you wanted people to help you with. Free advice from people donated in their own time, and now 22 emails in and you are getting snarky? Do you think that this is really going to encourage people to help you? Chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Thanks for the advice. All libraries are within the apps WEB-INF folder. It also doesn't appear to be a memory leak. Typically I would expect memory use to increase over time with a memory leak, but our app shows no increase just a sharp spike to max allocated memory prior to becoming unresponsive. For persistence we use Mybatis ORM which we have no issues with on other apps. I've set the GC logging up, but because there are no OOME no thread dumps are generated. Z. On 2/02/13 4:09 AM, Edson Richter edsonrich...@hotmail.com wrote: Em 01/02/2013 15:03, Edson Richter escreveu: Removing the hardware issues (faulty memory or disk), that you obviously already tested, I'll try to give some directions for testing: a) Main cause of memory leaks are hard references in main class loader. This happens when you put all your libraries into $TOMCAT_HOME/lib. Try to move your libraries into WEB-INF/lib and check how does your app behave. b) Lots of people forget to correctly close external resources (files, tcp connections, jdbc resources). Check your source code using FindBugs. It is not perfect, but will give you lots of warnings if you run on risk of not correctly closing resources. Remember, for jdbc resources, you should close all result sets first, then all statements, then all connections (not all database drivers will release resultset resources on statement close!). c) Also, we see incorrect thread programming... When I say (we see..) I want to mean: I see lots of junior programmers doing the same mistake over and over... I don't know if this is your case, but I feel that worth to mention here. remember to have a way to signalize to your threads that the application is closing or reloading. In my apps, I have a LifeCicly listener that will notify all threads that they should shutdown immediately. If a thread is stuck using resources, it will remain with that forever... d) If your app is JDK6 compatible, give a try on JRockit VM (from Oracle), and the excellent JRockit Mission Control that helps you to identify problems in real time. e) Never store content in static classes. The references stay forever. If you are using JPA, let JPA implementation to handle the cache. Use Soft Weak or Hard Weak if using EclipseLink. f) Never ever use OneToMany just because it is easy. If you have one object that has OneToMany to other 100, that these has One to many to another 100, you will have 10001 objects in memory with 1 query that is supposed to returns 1 record (the first object). If your query returns 10 records of the first object, you would have 11 object in memory... If you have 20 users using different objects... well, you got the point, right? By using good server hardware (ECC memory, SCSI disks, etc), a stable linux distro (my preference is for CentOS 64bit), and following the rules above, I manage to have web apps that run withing 2Gb of memory (on 8Gb of hardware), hundred of users in databases with 20Gb, 24x7. I hope this helps, Edson Richter Em 31/01/2013 23:36, Zoran Avtarovski escreveu: Hi Guys, We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. We set setup monitoring using JavaMelody and what we see is dramatic spikes in CPU and memory usage at the time of the crash. Memory hovers around 3-5% for the rest of the time and CPU is the same. I've looked at the number of sessions, HTTP activity , jdbc activity and nothing obvious jumps out. I'd really appreciate your collective wisdom in putting into practice some strategies to identify the cause of the spikes. This driving me and my team nuts. Any help would be appreciated. Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Em 03/02/2013 01:35, Zoran Avtarovski escreveu: Thanks for the advice. All libraries are within the apps WEB-INF folder. It also doesn't appear to be a memory leak. Typically I would expect memory use to increase over time with a memory leak, but our app shows no increase just a sharp spike to max allocated memory prior to becoming unresponsive. Which VM are you using? I had similar problems (even Kernel Panic!) running OpenJDK (default in some Linux distros). Regards, Edson For persistence we use Mybatis ORM which we have no issues with on other apps. I've set the GC logging up, but because there are no OOME no thread dumps are generated. Z. On 2/02/13 4:09 AM, Edson Richter edsonrich...@hotmail.com wrote: Em 01/02/2013 15:03, Edson Richter escreveu: Removing the hardware issues (faulty memory or disk), that you obviously already tested, I'll try to give some directions for testing: a) Main cause of memory leaks are hard references in main class loader. This happens when you put all your libraries into $TOMCAT_HOME/lib. Try to move your libraries into WEB-INF/lib and check how does your app behave. b) Lots of people forget to correctly close external resources (files, tcp connections, jdbc resources). Check your source code using FindBugs. It is not perfect, but will give you lots of warnings if you run on risk of not correctly closing resources. Remember, for jdbc resources, you should close all result sets first, then all statements, then all connections (not all database drivers will release resultset resources on statement close!). c) Also, we see incorrect thread programming... When I say (we see..) I want to mean: I see lots of junior programmers doing the same mistake over and over... I don't know if this is your case, but I feel that worth to mention here. remember to have a way to signalize to your threads that the application is closing or reloading. In my apps, I have a LifeCicly listener that will notify all threads that they should shutdown immediately. If a thread is stuck using resources, it will remain with that forever... d) If your app is JDK6 compatible, give a try on JRockit VM (from Oracle), and the excellent JRockit Mission Control that helps you to identify problems in real time. e) Never store content in static classes. The references stay forever. If you are using JPA, let JPA implementation to handle the cache. Use Soft Weak or Hard Weak if using EclipseLink. f) Never ever use OneToMany just because it is easy. If you have one object that has OneToMany to other 100, that these has One to many to another 100, you will have 10001 objects in memory with 1 query that is supposed to returns 1 record (the first object). If your query returns 10 records of the first object, you would have 11 object in memory... If you have 20 users using different objects... well, you got the point, right? By using good server hardware (ECC memory, SCSI disks, etc), a stable linux distro (my preference is for CentOS 64bit), and following the rules above, I manage to have web apps that run withing 2Gb of memory (on 8Gb of hardware), hundred of users in databases with 20Gb, 24x7. I hope this helps, Edson Richter Em 31/01/2013 23:36, Zoran Avtarovski escreveu: Hi Guys, We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. We set setup monitoring using JavaMelody and what we see is dramatic spikes in CPU and memory usage at the time of the crash. Memory hovers around 3-5% for the rest of the time and CPU is the same. I've looked at the number of sessions, HTTP activity , jdbc activity and nothing obvious jumps out. I'd really appreciate your collective wisdom in putting into practice some strategies to identify the cause of the spikes. This driving me and my team nuts. Any help would be appreciated. Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
I'm using the Sun JVM 1.6.0_31-b04. We had other performance issues with openJDK. Z. On 3/02/13 2:41 PM, Edson Richter edsonrich...@hotmail.com wrote: Em 03/02/2013 01:35, Zoran Avtarovski escreveu: Thanks for the advice. All libraries are within the apps WEB-INF folder. It also doesn't appear to be a memory leak. Typically I would expect memory use to increase over time with a memory leak, but our app shows no increase just a sharp spike to max allocated memory prior to becoming unresponsive. Which VM are you using? I had similar problems (even Kernel Panic!) running OpenJDK (default in some Linux distros). Regards, Edson For persistence we use Mybatis ORM which we have no issues with on other apps. I've set the GC logging up, but because there are no OOME no thread dumps are generated. Z. On 2/02/13 4:09 AM, Edson Richter edsonrich...@hotmail.com wrote: Em 01/02/2013 15:03, Edson Richter escreveu: Removing the hardware issues (faulty memory or disk), that you obviously already tested, I'll try to give some directions for testing: a) Main cause of memory leaks are hard references in main class loader. This happens when you put all your libraries into $TOMCAT_HOME/lib. Try to move your libraries into WEB-INF/lib and check how does your app behave. b) Lots of people forget to correctly close external resources (files, tcp connections, jdbc resources). Check your source code using FindBugs. It is not perfect, but will give you lots of warnings if you run on risk of not correctly closing resources. Remember, for jdbc resources, you should close all result sets first, then all statements, then all connections (not all database drivers will release resultset resources on statement close!). c) Also, we see incorrect thread programming... When I say (we see..) I want to mean: I see lots of junior programmers doing the same mistake over and over... I don't know if this is your case, but I feel that worth to mention here. remember to have a way to signalize to your threads that the application is closing or reloading. In my apps, I have a LifeCicly listener that will notify all threads that they should shutdown immediately. If a thread is stuck using resources, it will remain with that forever... d) If your app is JDK6 compatible, give a try on JRockit VM (from Oracle), and the excellent JRockit Mission Control that helps you to identify problems in real time. e) Never store content in static classes. The references stay forever. If you are using JPA, let JPA implementation to handle the cache. Use Soft Weak or Hard Weak if using EclipseLink. f) Never ever use OneToMany just because it is easy. If you have one object that has OneToMany to other 100, that these has One to many to another 100, you will have 10001 objects in memory with 1 query that is supposed to returns 1 record (the first object). If your query returns 10 records of the first object, you would have 11 object in memory... If you have 20 users using different objects... well, you got the point, right? By using good server hardware (ECC memory, SCSI disks, etc), a stable linux distro (my preference is for CentOS 64bit), and following the rules above, I manage to have web apps that run withing 2Gb of memory (on 8Gb of hardware), hundred of users in databases with 20Gb, 24x7. I hope this helps, Edson Richter Em 31/01/2013 23:36, Zoran Avtarovski escreveu: Hi Guys, We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. We set setup monitoring using JavaMelody and what we see is dramatic spikes in CPU and memory usage at the time of the crash. Memory hovers around 3-5% for the rest of the time and CPU is the same. I've looked at the number of sessions, HTTP activity , jdbc activity and nothing obvious jumps out. I'd really appreciate your collective wisdom in putting into practice some strategies to identify the cause of the spikes. This driving me and my team nuts. Any help would be appreciated. Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional
Re: Help in diagnosing server unresponsiveness
Thanks for the reply Chris, The crash is more a hang than a crash as we don't get OOME. The app just stops responding and the monitoring data shows spikes in memory and CPU to 100% and then nothing. I can't find anything in the linux system logs or in the tomcat or java logs. My hope is that once I can work out what's happening I can then fix the cause, but at the moment I'm flying blind on this. Z. On 2/02/13 6:08 AM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Zoran, On 1/31/13 8:36 PM, Zoran Avtarovski wrote: We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. Can you describe the crash in more detail? OOME? IF so, what kind (heap or PermGen)? Lock-up (deadlock, etc)? Actual JVM crash (produces a core dump or native stack dump)? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlEMEpAACgkQ9CaO5/Lv0PDmvgCeL+F9onCbvkWL5+1BEhAGcoY5 mxYAnif8os4/JiNaH2U5OmiWYB/GXMKm =2pXj -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Hi Edson, We do have some background threads as we use Quartz for scheduling tasks but we haven't had any issues with it in the past. I also checked the monitoring and I'm seeing anything strange during the execution of the scheduled tasks. Z. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Em 03/02/2013 02:01, Zoran Avtarovski escreveu: Hi Edson, We do have some background threads as we use Quartz for scheduling tasks but we haven't had any issues with it in the past. I also checked the monitoring and I'm seeing anything strange during the execution of the scheduled tasks. Z. My best advice is to put your application in executing using some monitoring tool (like the JRockit I proposed before, with the Mission Control) to try to discover the root of the problem. Regards, Edson - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Thanks Miguel, This is what I also suspect, but I can't see any evidence. The server has gone 10 days under heavy loads without a glitch and then it will hang a couple of times in the next few days with no apparent rhyme or reason. Z. On 3/02/13 5:56 AM, Miguel González Castaños miguel_3_gonza...@yahoo.es wrote: On 01/02/2013 20:08, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Zoran, On 1/31/13 8:36 PM, Zoran Avtarovski wrote: We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. Can you describe the crash in more detail? OOME? IF so, what kind (heap or PermGen)? Lock-up (deadlock, etc)? Actual JVM crash (produces a core dump or native stack dump)? I would go in that direction too. Enable logs and core or stack dumps and analyze them. Be sure you are not restarting Tomcat in your crontab (i had a server which was restarted once a week and masked some memory starvation). In my case I can tell you I had to end up disabling JaveMelody (it was provoking some side effects in our webapp as not managing international chars in the right way). If reports from Javamelody are not giving you any clue, beware that Javamelody has its own memory overhead (not much in your case but in my case it was around 200 Mb of heap in a 1 Gb virtual server). I followed Chris directions, I got stack dumps after a server crash and analyzed it with eclipse analyzer. I realized our programmer decided to load too many objects in memory than the server could cope with. So no memory leak but bad memory management was the root cause. Regards, Miguel - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Hi Howard, The move to linux was part of a move in-house for our client as the web services are only accessible behind the firewall. My gut feeling is that the issue isn't related to the WS as they run on a scheduled task 3 times a day. I think the issue lies in our app and struggling with not being able to see exactly what's happening during the crash. JavaMelody provides some insight but just not enough. I'm quite happy to post the charts for others to see. Just not sure what the best way to do it is. Z. On 3/02/13 3:11 PM, Howard W. Smith, Jr. smithh032...@gmail.com wrote: I know this is asking for too much or might be impossible to do but process of elimination.. If it was possible to eliminate or prevent web services from executing or being accessed, and no spikes occur, then problem is there. I think you said earlier that system was stable on Windows and migration to Linux was driven by the web services requirement. I wonder what kind of processing in those web services which may be causing this. A lot of database access, even more database access now because of web services? Did some developer try to add a manual call to gc, somewhere in the app to free resources. Maybe you can poll any / all developers or search code accordingly. Does the spike occur at certain time of day, maybe some code executed on schedule, or does it occur after certain activity occur in the app either by endusers or background processing? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
Just occurred to me: Linux has one feature that Windows doesn't: the OOM killer. When happens, normally you get a message in log. Have you checked that? If not, there is a little introduction here: http://linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html?page=1 Also, I had some issues with OOM killer in virtual machines running over qemu (actually, the OOM was happening in the host machine, not in guest - then in guest, had no error message). Regards, Edson Em 03/02/2013 02:07, Zoran Avtarovski escreveu: Thanks Miguel, This is what I also suspect, but I can't see any evidence. The server has gone 10 days under heavy loads without a glitch and then it will hang a couple of times in the next few days with no apparent rhyme or reason. Z. On 3/02/13 5:56 AM, Miguel González Castaños miguel_3_gonza...@yahoo.es wrote: On 01/02/2013 20:08, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Zoran, On 1/31/13 8:36 PM, Zoran Avtarovski wrote: We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. Can you describe the crash in more detail? OOME? IF so, what kind (heap or PermGen)? Lock-up (deadlock, etc)? Actual JVM crash (produces a core dump or native stack dump)? I would go in that direction too. Enable logs and core or stack dumps and analyze them. Be sure you are not restarting Tomcat in your crontab (i had a server which was restarted once a week and masked some memory starvation). In my case I can tell you I had to end up disabling JaveMelody (it was provoking some side effects in our webapp as not managing international chars in the right way). If reports from Javamelody are not giving you any clue, beware that Javamelody has its own memory overhead (not much in your case but in my case it was around 200 Mb of heap in a 1 Gb virtual server). I followed Chris directions, I got stack dumps after a server crash and analyzed it with eclipse analyzer. I realized our programmer decided to load too many objects in memory than the server could cope with. So no memory leak but bad memory management was the root cause. Regards, Miguel - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
On 2/2/2013 8:07 PM, Zoran Avtarovski wrote: Thanks Miguel, This is what I also suspect, but I can't see any evidence. The server has gone 10 days under heavy loads without a glitch and then it will hang a couple of times in the next few days with no apparent rhyme or reason. Z. On 3/02/13 5:56 AM, Miguel González Castaños miguel_3_gonza...@yahoo.es wrote: On 01/02/2013 20:08, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Zoran, On 1/31/13 8:36 PM, Zoran Avtarovski wrote: We have a application running on the latest Tomcat7 and we are getting a server crash or becoming unresponsive. This occur every few days at no fixed intervals or time of day and they certainly don't correlate to any app function at least not according to the logs. Can you describe the crash in more detail? OOME? IF so, what kind (heap or PermGen)? Lock-up (deadlock, etc)? Actual JVM crash (produces a core dump or native stack dump)? I would go in that direction too. Enable logs and core or stack dumps and analyze them. Be sure you are not restarting Tomcat in your crontab (i had a server which was restarted once a week and masked some memory starvation). In my case I can tell you I had to end up disabling JaveMelody (it was provoking some side effects in our webapp as not managing international chars in the right way). If reports from Javamelody are not giving you any clue, beware that Javamelody has its own memory overhead (not much in your case but in my case it was around 200 Mb of heap in a 1 Gb virtual server). I followed Chris directions, I got stack dumps after a server crash and analyzed it with eclipse analyzer. I realized our programmer decided to load too many objects in memory than the server could cope with. So no memory leak but bad memory management was the root cause. Regards, Miguel I've sort of followed this thread. If I remember correctly, you've recently moved to Linux. Here's an approach that might tell you what's going on at the time of the problem. When you're experiencing the problem, if you can, get a full thread dump. For Linux, that means you send a -3 signal to the PID (kill -3 PID). PID is the process ID of the Tomcat in trouble. At the same time, either run ps -FL PID or run top -p PID and switch to watching threads (H). See if you can see which thread is using 100% of the CPU from either PS or top. Sadly, the thread ID in the thread dump is hex, while the thread ID in either of the above two methods is decimal. So you'll have to do a bit of digging. The number in the thread dump is nid=0xNNN. In top, the thread ID will be displayed as the process ID. In ps, the thread ID will be displayed as the LWP ID. Hopefully, if you catch the Tomcat process at the right time, you'll be able to see which thread is consuming all the CPU, and from the thread dump see what the thread is doing. . . . . just my (Saturday night) two cents /mde/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Help in diagnosing server unresponsiveness
On 03/02/2013 3:17 PM, Zoran Avtarovski zo...@sparecreative.com wrote: Hi Howard, The move to linux was part of a move in-house for our client as the web services are only accessible behind the firewall. My gut feeling is that the issue isn't related to the WS as they run on a scheduled task 3 times a day. I think the issue lies in our app and struggling with not being able to see exactly what's happening during the crash. JavaMelody provides some insight but just not enough. Norhing in the catalina.out or app specific logs? I'm quite happy to post the charts for others to see. Just not sure what the best way to do it is. Z. On 3/02/13 3:11 PM, Howard W. Smith, Jr. smithh032...@gmail.com wrote: I know this is asking for too much or might be impossible to do but process of elimination.. If it was possible to eliminate or prevent web services from executing or being accessed, and no spikes occur, then problem is there. I think you said earlier that system was stable on Windows and migration to Linux was driven by the web services requirement. I wonder what kind of processing in those web services which may be causing this. A lot of database access, even more database access now because of web services? Did some developer try to add a manual call to gc, somewhere in the app to free resources. Maybe you can poll any / all developers or search code accordingly. Does the spike occur at certain time of day, maybe some code executed on schedule, or does it occur after certain activity occur in the app either by endusers or background processing? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
