Re: Tomcat Shutting Down by Itself?
On 28.07.2010 01:01, Robinson, Eric wrote: 2) Use System.getRuntime().addShutdownHook() to trigger your own thread when the JVM does decide to exit. Assuming the OP does not have easy access to the running application, would that mean adding another application which is launched at Tomcat start, and runs the above ? Or do you need to do this in the application itself ? That was my next question, too. Sorry I don't remember the whole long discussion thread. So we definiteley know the process is gone and not that it is only no longer responding? Do we know whether it was an orderly Tomcat shutdown or not? Log messages in the Tomcat log files? Assuming it was not: do we know it was an orderly JVM shutdown. Here we could use the shutdown hook. You can - deploy another webapp (needs write access to Tomcat config or deployment area depending on config) - change the existing webapp (i.e. adding a context listner that registers the hook during startup; don't need to change existing webapp classes) - add a JSP which registers the hook when called (only needs write access to some content directory on the existing webapp) - Download TC source and add it there (if you can't change anything in the webapp or config directories, but can write to the Tomcat classes or lib directories. Or: do you see any hotspot error files, which should be there if the JVM crashes and can write to the working directory.If you had a newer JVM (recent 1.5 or 1.6) you could add the flag -XX:OnError=... to run a command at the moment it crashes. Concerning special requests which might trigger functionality leading to a crash: You could add a filter to the webapp doing a simplified access log in front of request processing and then check after crsh, which of those were missing in the final access log. To make matching easir, the filter could add a request attribute with a unique ID to each request and log it, and you can configure your normal access log pattern to also include this attribute. So you only have to compare the two ID lists to find the reuests, that entered Tomcat but were not finished when it crashed. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat Shutting Down by Itself?
So we definitely know the process is gone and not that it is only no longer responding? I'm not 100% certain. I did... netstat -an|grep :3057|grep LISTEN ..and nothing came back. From that we at least know that the process was no longer listening on its assigned port. Then I *think* I did a 'ps' and the java process was actually not there, but now I'm not sure. Do we know whether it was an orderly Tomcat shutdown or not? Log messages in the Tomcat log files? I don't see anything in the logs other than the shutdown message I posted at the beginning of this thread. deploy another webapp (needs write access to Tomcat config or deployment area depending on config) add a JSP which registers the hook when called (only needs write access to some content directory on the existing webapp) Those seem like my best two choices. -- Eric Robinson Disclaimer - July 28, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Shutting Down by Itself?
try to change shutdown port in the server.xml -- View this message in context: http://old.nabble.com/Tomcat-Shutting-Down-by-Itself--tp29249881p29293203.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Shutting Down by Itself?
Robinson, Eric wrote: What kind of machine are you running these 163/164 instances of Tomcat on, and when you are running them, what /does/ free say ? I have two different servers with 164 instances of tomcat. Both servers have 2x quad-core 2.8Ghz Xeon processors with 32GB RAM. On the first server (app03), most instances of tomcat are configured with 64MB of Java heap. About 20% of them have 96-256MB. I almost never reboot this server (current uptime 61 days). Here's 'free' from app03. [r...@app03 ~]# free total used free sharedbuffers cached Mem: 33265832 305702602695572 0 296976 4562784 -/+ buffers/cache: 257105007555332 Swap: 2031608 02031608 On the other server (app04), all instances of tomcat are configured with 512MB Java heap (-ms512M -mx512M). After 4 or 5 days of uptime, the server starts to swap a little. Then I reboot it and it is fine for several more days. As you can see from the following, it is about time for a reboot. If I do not reboot it tonight, by tomorrow or the next day it may be up to 1-2GB of swap. (It actually doesn't slow the server down much though. sar shows that it runs about 90% idle anyway, including iowait. [r...@app03 ~]# ssh app04 free total used free sharedbuffers cached Mem: 33265832 32965812 300020 0 191248 3842092 -/+ buffers/cache: 28932472460 Swap: 2031608 42882027320 That is very interesting information, thank you. Intuitively, I would have never thought that it was /possible/ to run 160 or so instances of Tomcat on anything but some type of very expensive hardware. Granted, dual-Xeon CPU mainboards with 32 GB are not the low-end, but still. One tends to forget how many megabytes fit in one gigabyte, and 160 X 64 MB is actually only 10 GB. The data above, and your comments, seem to make unlikely the OOM hypothesis, unless you have one application suddenly acting up. On which of those servers is the one Tomcat instance dying ? on both ? Is that Tomcat instance running an application which the other ones don't ? What kind of memory usage do the Tomcat instances have (in total, including the Heap) ? Do you start them with any kind of other memory-related parameter for the JVM apart from -Xms and -Xmx ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat Shutting Down by Itself?
Intuitively, I would have never thought that it was /possible/ to run 160 or so instances of Tomcat on anything but some type of very expensive hardware. I assure you that it works admirably! And each of the instances runs a sophisticated medical application with hundreds of JSPs and class files. The data above, and your comments, seem to make unlikely the OOM hypothesis, unless you have one application suddenly acting up. Also, we run a script that checks the logs for OOM errors and those are not showing up. On which of those servers is the one Tomcat instance dying ? on both ? Both occasionally. But now that we're digging into it, it has not happened for several days on either one. :-/ Is that Tomcat instance running an application which the other ones don't ? They all run copies of the same application, albeit slightly different versions. But the one that is shutting down is running the same version as other instances that are not shutting down. What kind of memory usage do the Tomcat instances have (in total, including the Heap) ? Depends on which server and when you check. Right now on app03: [r...@app03 ~]# ps axo rss,vsz,comm,pid|grep java|sort -g 44984 251104 java 9151 47972 251876 java10753 50816 253200 java19077 59540 253160 java17079 61780 254840 java12740 73196 244908 java26675 76580 252984 java29065 78224 262196 java 981 89080 255308 java17223 93968 255196 java13046 98272 256228 java17481 99440 256012 java25020 101744 264704 java9529 106160 258664 java 19425 108476 257452 java 26549 109712 266980 java8853 113688 261296 java 14086 117596 270040 java 26159 118356 269264 java8912 121316 270460 java 10800 123248 263196 java5202 125764 257728 java 28540 126544 258128 java7774 128084 267096 java 25420 128448 267420 java 13290 128508 259092 java 29286 129588 258800 java 21942 130068 265700 java 12465 130924 260472 java 29974 131484 268980 java8812 132208 261088 java7585 132560 268264 java 17343 133632 264320 java 16501 134232 261116 java 29183 135548 269132 java 10609 136932 273096 java 28742 140224 273052 java 12246 140624 274484 java 15351 141780 263020 java 12765 144948 264512 java9292 148512 276672 java9300 148900 267820 java9547 150448 724588 java 12883 152224 270336 java 13083 153040 271260 java9236 153300 293472 java9634 153648 276708 java 31556 154324 272836 java9041 154520 265512 java9894 155176 272788 java 11453 155940 271532 java 10843 155940 274996 java 15897 156920 273980 java8772 157240 271088 java 28906 157596 273128 java 16910 158852 275100 java 16096 160516 267404 java8813 161060 279840 java 11403 161840 277244 java 11150 162176 274532 java 11876 162508 271484 java9341 162672 274292 java 13402 162960 274220 java 10226 163460 279256 java 25700 163840 282796 java9988 164248 277304 java 18237 164800 277576 java 26965 164928 278300 java 32556 164992 274972 java 11334 165284 282936 java9587 165608 275548 java6217 165992 277956 java 24426 166180 271832 java 18542 166548 274660 java 10662 167004 277668 java 10179 167064 284676 java9386 167656 278904 java 12672 167712 278332 java6112 167716 278484 java 715 168052 278188 java3376 168224 274312 java2910 168720 277072 java 32519 170104 274900 java7194 170864 278304 java 16399 171036 275596 java 10375 171392 279968 java 12195 171884 281372 java 11207 172716 276524 java 23927 173192 279232 java2755 173636 279772 java 11569 173688 278308 java 18800 174560 277796 java 10721 174660 279300 java 31576 175448 278176 java 12907 175784 287360 java9198 176632 281812 java 15212 177588 293016 java 11750 178272 282860 java 19846 178704 280868 java 12023 179476 283092 java 28283 182580 283768 java 18964 183496 284244 java 15746 184928 282928 java9942 185100 287952 java 30676 186124 28 java 16051 186312 294096 java9100 187360 305840 java 29874 190032 303316 java 11266 190616 292284
Re: Tomcat Shutting Down by Itself?
Robinson, Eric wrote: Intuitively, I would have never thought that it was /possible/ to run 160 or so instances of Tomcat on anything but some type of very expensive hardware. I assure you that it works admirably! And each of the instances runs a sophisticated medical application with hundreds of JSPs and class files. The data above, and your comments, seem to make unlikely the OOM hypothesis, unless you have one application suddenly acting up. Also, we run a script that checks the logs for OOM errors and those are not showing up. On which of those servers is the one Tomcat instance dying ? on both ? Both occasionally. But now that we're digging into it, it has not happened for several days on either one. :-/ Is that Tomcat instance running an application which the other ones don't ? They all run copies of the same application, albeit slightly different versions. But the one that is shutting down is running the same version as other instances that are not shutting down. What kind of memory usage do the Tomcat instances have (in total, including the Heap) ? Depends on which server and when you check. Right now on app03: [r...@app03 ~]# ps axo rss,vsz,comm,pid|grep java|sort -g 44984 251104 java 9151 47972 251876 java10753 50816 253200 java19077 59540 253160 java17079 61780 254840 java12740 73196 244908 java26675 76580 252984 java29065 78224 262196 java 981 89080 255308 java17223 93968 255196 java13046 98272 256228 java17481 99440 256012 java25020 101744 264704 java9529 106160 258664 java 19425 108476 257452 java 26549 109712 266980 java8853 113688 261296 java 14086 117596 270040 java 26159 118356 269264 java8912 121316 270460 java 10800 123248 263196 java5202 125764 257728 java 28540 126544 258128 java7774 128084 267096 java 25420 128448 267420 java 13290 128508 259092 java 29286 129588 258800 java 21942 130068 265700 java 12465 130924 260472 java 29974 131484 268980 java8812 132208 261088 java7585 132560 268264 java 17343 133632 264320 java 16501 134232 261116 java 29183 135548 269132 java 10609 136932 273096 java 28742 140224 273052 java 12246 140624 274484 java 15351 141780 263020 java 12765 144948 264512 java9292 148512 276672 java9300 148900 267820 java9547 150448 724588 java 12883 152224 270336 java 13083 153040 271260 java9236 153300 293472 java9634 153648 276708 java 31556 154324 272836 java9041 154520 265512 java9894 155176 272788 java 11453 155940 271532 java 10843 155940 274996 java 15897 156920 273980 java8772 157240 271088 java 28906 157596 273128 java 16910 158852 275100 java 16096 160516 267404 java8813 161060 279840 java 11403 161840 277244 java 11150 162176 274532 java 11876 162508 271484 java9341 162672 274292 java 13402 162960 274220 java 10226 163460 279256 java 25700 163840 282796 java9988 164248 277304 java 18237 164800 277576 java 26965 164928 278300 java 32556 164992 274972 java 11334 165284 282936 java9587 165608 275548 java6217 165992 277956 java 24426 166180 271832 java 18542 166548 274660 java 10662 167004 277668 java 10179 167064 284676 java9386 167656 278904 java 12672 167712 278332 java6112 167716 278484 java 715 168052 278188 java3376 168224 274312 java2910 168720 277072 java 32519 170104 274900 java7194 170864 278304 java 16399 171036 275596 java 10375 171392 279968 java 12195 171884 281372 java 11207 172716 276524 java 23927 173192 279232 java2755 173636 279772 java 11569 173688 278308 java 18800 174560 277796 java 10721 174660 279300 java 31576 175448 278176 java 12907 175784 287360 java9198 176632 281812 java 15212 177588 293016 java 11750 178272 282860 java 19846 178704 280868 java 12023 179476 283092 java 28283 182580 283768 java 18964 183496 284244 java 15746 184928 282928 java9942 185100 287952 java 30676 186124 28 java 16051 186312 294096 java9100 187360 305840 java 29874 190032 303316 java
RE: Tomcat Shutting Down by Itself?
364600 491072 java5101 --Here's the occasional bad boy 604780 746120 java 31131 --Here's the occasional bad boy. Those sizes appear to be in Kbyte, so we are talking in one case of a process with a resident size of 360 MB, in the other of 600 MB.I guess thus, that these are not instances of JVM started with -Xms64M and -Xmx64M ? That's correct. This instance is started with '-ms256M -mx256M' on app03 and '-ms512M -mx512M' on app04. Mmm. So these instances are running the same application and version of ditto, on different hosts, and they both crash sometimes. But the same application/version is running in other instances, which do not crash. Exactly. Are the ones that crash used by the same users on both instances? Yes. The instances are behind a load balancer that uses a weighted least connection algorithm with 6 minutes of session persistence based on source IP. So the initial connection from any workstation goes to the server that is currently serving the fewest connections, but then subsequent connections from the same workstation go to the same server (unless it goes down or 6 idle minutes elapse). and are these users different ones than the ones using the other instances of the app that never crash ? Yes. Each instance serves a different set of users in a local office somewhere. -- Eric Robinson Disclaimer - July 27, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat Shutting Down by Itself?
Yes. Each instance serves a different set of users in a local office somewhere. By that I mean each pair of load-balanced instances serves a set of users in a local office somewhere. -- Disclaimer - July 27, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Shutting Down by Itself?
Robinson, Eric wrote: Yes. Each instance serves a different set of users in a local office somewhere. By that I mean each pair of load-balanced instances serves a set of users in a local office somewhere. Then my suspicions would be as follows : this particular set of users of this complex application, uses a particular functionality of the application, used by no other set of users, and that particular functionality contains (or triggers) a bug that blows away the server. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Shutting Down by Itself?
André Warnier wrote: Robinson, Eric wrote: Yes. Each instance serves a different set of users in a local office somewhere. By that I mean each pair of load-balanced instances serves a set of users in a local office somewhere. Then my suspicions would be as follows : this particular set of users of this complex application, uses a particular functionality of the application, used by no other set of users, and that particular functionality contains (or triggers) a bug that blows away the server. And we have a set of options here : - tell the customer to close down that office and move the people somewhere else, to a Tomcat instance that works. I they also then crash that one, then we know - more efficiently : switch the ports between those instances, and some other set of instances which at the moment do not crash. If the other one then starts crashing, then we also know. (Of course that may not work if each instance has its particular set of back-end data. That would be messy.) - or else, start capturing the packets destined to these crashing instances of Tomcat. Since they have their own port, it should be easy to do that with wireshark, filtering on the destination port. The next time it crashes, examine the end of the packet capture to see if you can determine the sequence which crashes the server. - or maybe simpler : when one of these instances crashes, do the users notice anything ? or does it just smoothly switch to the other balanced instance ? If they notice, they may be able to tell what they were doing just before ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat Shutting Down by Itself?
this particular set of users of this complex application, uses a particular functionality of the application, used by no other set of users, and that particular functionality contains (or triggers) a bug that blows away the server. That seems like the reasonable conclusion. The next question is whether there is a way to trap it and see what is happening. -- Eric Robinson Disclaimer - July 27, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Shutting Down by Itself?
On 27/07/2010 21:20, Robinson, Eric wrote: this particular set of users of this complex application, uses a particular functionality of the application, used by no other set of users, and that particular functionality contains (or triggers) a bug that blows away the server. That seems like the reasonable conclusion. The next question is whether there is a way to trap it and see what is happening. Access logs and look for a common pattern? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Shutting Down by Itself?
Mark Thomas wrote: On 27/07/2010 21:20, Robinson, Eric wrote: this particular set of users of this complex application, uses a particular functionality of the application, used by no other set of users, and that particular functionality contains (or triggers) a bug that blows away the server. That seems like the reasonable conclusion. The next question is whether there is a way to trap it and see what is happening. Access logs and look for a common pattern? That would be a possibility too, simple if it works. But access logs have a habit of being written at the end of a transaction, so in a case like this one, where the server just disapears without any evident trace, you may miss exactly the one that you are looking for. I don't know about Tomcat though. Would there be any specific way to catch a call to system.exit() ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat Shutting Down by Itself?
From: André Warnier [mailto:a...@ice-sa.com] Subject: Re: Tomcat Shutting Down by Itself? Would there be any specific way to catch a call to system.exit() ? 1) Enable a SecurityManager (usually a major pain). 2) Use System.getRuntime().addShutdownHook() to trigger your own thread when the JVM does decide to exit. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Shutting Down by Itself?
Caldarale, Charles R wrote: From: André Warnier [mailto:a...@ice-sa.com] Subject: Re: Tomcat Shutting Down by Itself? Would there be any specific way to catch a call to system.exit() ? 1) Enable a SecurityManager (usually a major pain). Because you would need to authorise everything else that this application /might/ be doing, except system.exit(), right ? 2) Use System.getRuntime().addShutdownHook() to trigger your own thread when the JVM does decide to exit. Assuming the OP does not have easy access to the running application, would that mean adding another application which is launched at Tomcat start, and runs the above ? Or do you need to do this in the application itself ? No other way from outside, apart from the SecurityManager ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat Shutting Down by Itself?
2) Use System.getRuntime().addShutdownHook() to trigger your own thread when the JVM does decide to exit. Assuming the OP does not have easy access to the running application, would that mean adding another application which is launched at Tomcat start, and runs the above ? Or do you need to do this in the application itself ? That was my next question, too. -- Eric Robinson Disclaimer - July 27, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat Shutting Down by Itself?
From: André Warnier [mailto:a...@ice-sa.com] Subject: Re: Tomcat Shutting Down by Itself? 1) Enable a SecurityManager (usually a major pain). Because you would need to authorise everything else that this application /might/ be doing, except system.exit(), right ? Correct. Getting the settings right might involve much experimentation and a few application errors along the way - it all depends on what the webapp needs to do. 2) Use System.getRuntime().addShutdownHook() to trigger your own thread when the JVM does decide to exit. would that mean adding another application which is launched at Tomcat start, and runs the above ? Yes, that would probably be the easiest way. Or do you need to do this in the application itself ? No, since the shutdown hook applies to the JVM, not individual webapps. No other way from outside, apart from the SecurityManager ? Not that I can think of right off the bat. Others might know easier tricks specific to Tomcat. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat Shutting Down by Itself?
What I was trying to say above, in a devious but humoristic way, is that there are not a lot of people on this list running those versions anymore. Fair enough. I realize it's old, but it still works great for 163 of the 164 instances on this server. However, an upgrade is coming soon and perhaps that will resolve it. Thanks. -- Eric Robinson Disclaimer - July 26, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Shutting Down by Itself?
Robinson, Eric wrote: What I was trying to say above, in a devious but humoristic way, is that there are not a lot of people on this list running those versions anymore. Fair enough. I realize it's old, but it still works great for 163 of the 164 instances on this server. However, an upgrade is coming soon and perhaps that will resolve it. Thanks. You are welcome. But you have made me curious. What kind of machine are you running these 163/164 instances of Tomcat on, and when you are running them, what /does/ free say ? (The first 10 lines of top with option m would be ok too) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat Shutting Down by Itself?
What kind of machine are you running these 163/164 instances of Tomcat on, and when you are running them, what /does/ free say ? I have two different servers with 164 instances of tomcat. Both servers have 2x quad-core 2.8Ghz Xeon processors with 32GB RAM. On the first server (app03), most instances of tomcat are configured with 64MB of Java heap. About 20% of them have 96-256MB. I almost never reboot this server (current uptime 61 days). Here's 'free' from app03. [r...@app03 ~]# free total used free sharedbuffers cached Mem: 33265832 305702602695572 0 296976 4562784 -/+ buffers/cache: 257105007555332 Swap: 2031608 02031608 On the other server (app04), all instances of tomcat are configured with 512MB Java heap (-ms512M -mx512M). After 4 or 5 days of uptime, the server starts to swap a little. Then I reboot it and it is fine for several more days. As you can see from the following, it is about time for a reboot. If I do not reboot it tonight, by tomorrow or the next day it may be up to 1-2GB of swap. (It actually doesn't slow the server down much though. sar shows that it runs about 90% idle anyway, including iowait. [r...@app03 ~]# ssh app04 free total used free sharedbuffers cached Mem: 33265832 32965812 300020 0 191248 3842092 -/+ buffers/cache: 28932472460 Swap: 2031608 42882027320 -- Eric Robinson Disclaimer - July 26, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat Shutting Down by Itself?
From: Robinson, Eric [mailto:eric.robin...@psmnv.com] Subject: Tomcat Shutting Down by Itself? We've observed tomcat5 Tomcat 5.x.y what? Be specific. Also tell us the JVM you're using and the platform you're running on. shutting down seemingly by itself recently. This morning I saw this in the log: - Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutting down. There's nothing in the above that indicates Tomcat is shutting down (Tomcat itself doesn't use Quartz). If one of your webapps is doing something antisocial such as calling System.exit(), that will cause a shutdown. What else is in the logs? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Shutting Down by Itself?
On 23.07.2010 19:43, Robinson, Eric wrote: We've observed tomcat5 shutting down seemingly by itself recently. This morning I saw this in the log: - Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutting down. Any way to tell why tomcat is shutting down? Maybe something from this recent discussion applies: http://marc.info/?t=12767355721r=1w=2 Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat Shutting Down by Itself?
Tomcat 5.x.y what? Be specific. Also tell us the JVM you're using and the platform you're running on. Apache Tomcat/5.0.28 Java 1.4.2_09-b05 Sun Microsystems Inc. Linux 2.6.18-92.1.22.el5PAE i386 If one of your webapps is doing something antisocial such as calling System.exit(), that will cause a shutdown. Not too likely as there are 100+ separate instances of tomcat on the same server listening on different ports and running the same webapp from differenet directories. The other instances are fine and have been for years. This is the only instance that has the problem and it has only been happening for a few weeks. What else is in the logs? Not much except some errors related to date parsing (an obvious coding bug) and a socket connection timeout related to faxing. java.lang.NullPointerException at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1108) at java.text.DateFormat.parse(DateFormat.java:333) at catalog.EcwFax.getEasternTime(EcwFax.java:2281) at catalog.EcwFax.LogFaxRequest(EcwFax.java:887) at catalog.EcwFax.LogFaxRequest(EcwFax.java:790) at org.apache.jsp.jsp.catalog.xml.fax.LogFaxRequest_jsp._jspService(LogFaxR equest_jsp.java:142) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:94) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.ja va:324) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:292) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:236) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica tionFilterChain.java:237) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt erChain.java:157) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv e.java:214) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveCo ntext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:5 20) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardCon textValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv e.java:152) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveCo ntext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:5 20) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java :137) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveCo ntext.java:104) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java :118) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveCo ntext.java:102) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:535 ) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveCo ntext.java:102) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:5 20) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve. java:109) at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveCo ntext.java:104) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:5 20) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:79 9) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processC onnection(Http11Protocol.java:705) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:57 7) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool java:683) at java.lang.Thread.run(Thread.java:534) java.lang.NullPointerException at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1108) at java.text.DateFormat.parse(DateFormat.java:333) at catalog.EcwFax.getEasternTime(EcwFax.java:2281) at catalog.EcwFax.LogFaxRequest(EcwFax.java:887) at catalog.EcwFax.LogFaxRequest(EcwFax.java:790) at org.apache.jsp.jsp.catalog.xml.fax.LogFaxRequest_jsp._jspService(LogFaxR equest_jsp.java:142) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:94) at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.ja va:324) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:292) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:236) at
RE: Tomcat Shutting Down by Itself?
Maybe something from this recent discussion applies: http://marc.info/?t=12767355721r=1w=2 Good thread, thanks. Unfortunately, the situations it describes don't apply in this case. -- Eric Robinson Disclaimer - July 23, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Shutting Down by Itself?
Robinson, Eric wrote: Tomcat 5.x.y what? Be specific. Also tell us the JVM you're using and the platform you're running on. Apache Tomcat/5.0.28 Java 1.4.2_09-b05 Sun Microsystems Inc. Linux 2.6.18-92.1.22.el5PAE i386 Gentlemen, I think we need a minute of silence here, and a golden watch. The above versions of Java and Tomcat are so old, that it is probably the explanation. This Tomcat and Java are just trying to say, in the only way they can, that after some 10 years of good and hard labor, it is now time to retire them and have the burden taken over by a new generation. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat Shutting Down by Itself?
This Tomcat and Java are just trying to say, in the only way they can, that after some 10 years of good and hard labor, it is now time to retire them and have the burden taken over by a new generation. Yes, that's probably it. One of the... [r...@app03 ~]# ps ax|grep java|wc -l 164 ..164 instances of tomcat that are running on this server is bored with being old and has decided to act up. The other 163 instances are lame for insisting on working fine day after day. :-) -- Eric Robinson Disclaimer - July 23, 2010 This email and any files transmitted with it are confidential and intended solely for Tomcat Users List. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of . Warning: Although has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Shutting Down by Itself?
Robinson, Eric wrote: This Tomcat and Java are just trying to say, in the only way they can, that after some 10 years of good and hard labor, it is now time to retire them and have the burden taken over by a new generation. Yes, that's probably it. One of the... [r...@app03 ~]# ps ax|grep java|wc -l 164 ..164 instances of tomcat that are running on this server is bored with being old and has decided to act up. The other 163 instances are lame for insisting on working fine day after day. :-) Out of memory ? (which is also something that happens frequently to old people, and to java programs) What I was trying to say above, in a devious but humoristic way, is that there are not a lot of people on this list running those versions anymore. There may not even be a lot of people on this list with any experience of running that anymore. Probably, any people with the experience of running these versions have been since promoted and don't have the time to be on this list anymore. So you are basically stuck with Chuck and me, and we are telling you that Tomcat does not shut down by itself. I'll add that when it shuts down properly, it usually says so in its logfiles. If it doesn't, then it must be that it is being brutally terminated (or euthanasied ?) by something else. Under Linux, there is some talk of an OOM killer character who has been known to do that kind of thing. When all these JVM instances are running, what does free say ? And while you're at it, go to the Tomcat bin subdirectory, and tell us what version.sh has to say. Another possibility on a Linux system, is that some smart /etc/init.d startup script for Tomcat redirects Tomcat's messages somewhere else, like /var/log/daemon.log. Have you been looking there ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
