Re: tomcat 7.0.40 - http sessions not expiring
Hi Chris, Thanks for the help. I don't see any OOME around this time. Before restarting I collected the heap and thread dumps. Looking at one of the thread dumps, I see the ContainerBackgroundProcessor (below). So looks like that thread didn't die entirely, may be it just stopped working ? *"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x7fd000a5d800 nid=0x9be5 waiting on condition [0x7fcfc4955000]* * java.lang.Thread.State: TIMED_WAITING (sleeping)* * at java.lang.Thread.sleep(Native Method)* * at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1508)* * at java.lang.Thread.run(Thread.java:745)* * Locked ownable synchronizers:* * - None* - I have attached a screenshot of the StandardManager runtime attributes from the heapdump. Is there anything anyone can see wrong here? - Is there anything else I can look for in the heapdump and threaddump? [image: Inline image 1] thanks, Stephen On Fri, Jun 3, 2016 at 5:57 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Stephen, > > On 6/3/16 8:47 PM, Stephen Bhadran wrote: > > *Linux version:* 2.6.32-279.5.2.el6.x86_64 ( > > mockbu...@c6b10.bsys.dev.centos.org) (gcc version 4.4.6 20120305 > > (Red Hat 4.4.6-4) (GCC) ) *Java Version: *1.7.0_55 *Tomcat > > Version: *7.0.40 *Session Timeout: *30 minutes > > > > Hi, Two of our servers reached max heap and we had to restart to > > resolve the issue. > > > > In troubleshooting, we noticed Http Sessions had stopped expiring > > 05/12th and reached 25K over a period of 11 days. I noticed the > > same behavior in both servers, the session expiration seem to have > > stopped on both of them around the same time. I'm saying this > > because from looking at the monitoring graphs (NewRelic) I notice > > the http sessions starting to climb on both servers around the same > > time frame. > > > > I am thinking, the issue could be two folds: #1 - The session > > expiration stopped working -OR #2 - The sessions themselves were > > created with TTLs way into the future > > > > I suspected #1. I looked that tomcat 7.0.40 source code, extracted > > all session/manager error messages and searched for them in the > > logs and didn't find any hits. I was hoping to find something about > > "Session expiration stopped working", but didn't find any. > > > > Can you please advise how to troubleshoot this issue? I'd very > > much appreciate the help. > > Were there any errors around the 12th? I've found that in rare cases, > the server hits an OOME while the BackgroundProcessor thread is > running, which kills the BackgroundProcessor. > > Guess which thread is responsible for triggering the session-cleaner? > > If the BackgroundProcessor died, it should have dumped an error to a > log file. As I said, it's pretty rare, but it does happen and then > it's only a matter of time before your sessions bust your heap. > > You can manually-trigger the session-cleaner via JMX of you absolutely > have to, but I'd personally prefer to bounce a Tomcat instance in that > case. > > - -chris > -BEGIN PGP SIGNATURE- > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIcBAEBCAAGBQJXUid5AAoJEBzwKT+lPKRYJfYQAKA2ditjB7B2vLbm0po83Tr0 > YD2EjqLBrF4O/3Wu6enIDRAQEIGgXmTr2bgPlMvu+6aoDopiCSKltRxWQFlTMyIP > 8ura/0YlS5sX3yGPPKN5nrZnOz2dvrgPncHLNVfuM9BEm+8QZPeKNZVDhMH53m1v > 4YcLCM5IjswAS4FrVeG9A0gcZplBd53WUuypnRYQMUcS1sACdItoZ3TLAwb/WBg/ > T9yoyftEXJVtP6lovkaWka3yw3v382P6HmSV6/TtCWjogB/tBdCsEPmGEQ0j7k8y > 9dLmmTBawDdckaxiGf1M1M/5R+f7XqU4e02Dnhk9v4umysmu5EH+VQFO4kfi5qX+ > DvaoS13lluyN1LGMyuwHkYFHXxlgZhR/XAW9Cn/0D/NDCQks6lHG9Sby+T/kvLSO > YNSBV78SX9wsJ/fH9MxrcTarxyUYsxnHdNkcarkmcrQIkhda/vBXQobXpYVL9poR > HmxNGNOujCTojqPJzlpgAaUVUq6pEUcN3u1vkCqXNQGVo983PHz2JhfEBGYihKBk > icbuFxfIy2Dv6KwuPPMV9Y7nVRNVBrVEpieZuAc7Sj9/U3kUgPUznUSbRFUCnGd1 > YI/kq9rn/qj/aqhaJdMF0Ci4+ieJMorKNUE1PWOjbnya8XoyBxXfPd5iV6n7elEw > V6qNyO6n46isrAiVEu5c > =MyIX > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: tomcat 7.0.40 - http sessions not expiring
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Stephen, On 6/3/16 8:47 PM, Stephen Bhadran wrote: > *Linux version:* 2.6.32-279.5.2.el6.x86_64 ( > mockbu...@c6b10.bsys.dev.centos.org) (gcc version 4.4.6 20120305 > (Red Hat 4.4.6-4) (GCC) ) *Java Version: *1.7.0_55 *Tomcat > Version: *7.0.40 *Session Timeout: *30 minutes > > Hi, Two of our servers reached max heap and we had to restart to > resolve the issue. > > In troubleshooting, we noticed Http Sessions had stopped expiring > 05/12th and reached 25K over a period of 11 days. I noticed the > same behavior in both servers, the session expiration seem to have > stopped on both of them around the same time. I'm saying this > because from looking at the monitoring graphs (NewRelic) I notice > the http sessions starting to climb on both servers around the same > time frame. > > I am thinking, the issue could be two folds: #1 - The session > expiration stopped working -OR #2 - The sessions themselves were > created with TTLs way into the future > > I suspected #1. I looked that tomcat 7.0.40 source code, extracted > all session/manager error messages and searched for them in the > logs and didn't find any hits. I was hoping to find something about > "Session expiration stopped working", but didn't find any. > > Can you please advise how to troubleshoot this issue? I'd very > much appreciate the help. Were there any errors around the 12th? I've found that in rare cases, the server hits an OOME while the BackgroundProcessor thread is running, which kills the BackgroundProcessor. Guess which thread is responsible for triggering the session-cleaner? If the BackgroundProcessor died, it should have dumped an error to a log file. As I said, it's pretty rare, but it does happen and then it's only a matter of time before your sessions bust your heap. You can manually-trigger the session-cleaner via JMX of you absolutely have to, but I'd personally prefer to bounce a Tomcat instance in that case. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJXUid5AAoJEBzwKT+lPKRYJfYQAKA2ditjB7B2vLbm0po83Tr0 YD2EjqLBrF4O/3Wu6enIDRAQEIGgXmTr2bgPlMvu+6aoDopiCSKltRxWQFlTMyIP 8ura/0YlS5sX3yGPPKN5nrZnOz2dvrgPncHLNVfuM9BEm+8QZPeKNZVDhMH53m1v 4YcLCM5IjswAS4FrVeG9A0gcZplBd53WUuypnRYQMUcS1sACdItoZ3TLAwb/WBg/ T9yoyftEXJVtP6lovkaWka3yw3v382P6HmSV6/TtCWjogB/tBdCsEPmGEQ0j7k8y 9dLmmTBawDdckaxiGf1M1M/5R+f7XqU4e02Dnhk9v4umysmu5EH+VQFO4kfi5qX+ DvaoS13lluyN1LGMyuwHkYFHXxlgZhR/XAW9Cn/0D/NDCQks6lHG9Sby+T/kvLSO YNSBV78SX9wsJ/fH9MxrcTarxyUYsxnHdNkcarkmcrQIkhda/vBXQobXpYVL9poR HmxNGNOujCTojqPJzlpgAaUVUq6pEUcN3u1vkCqXNQGVo983PHz2JhfEBGYihKBk icbuFxfIy2Dv6KwuPPMV9Y7nVRNVBrVEpieZuAc7Sj9/U3kUgPUznUSbRFUCnGd1 YI/kq9rn/qj/aqhaJdMF0Ci4+ieJMorKNUE1PWOjbnya8XoyBxXfPd5iV6n7elEw V6qNyO6n46isrAiVEu5c =MyIX -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
tomcat 7.0.40 - http sessions not expiring
*Linux version:* 2.6.32-279.5.2.el6.x86_64 ( mockbu...@c6b10.bsys.dev.centos.org) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) *Java Version: *1.7.0_55 *Tomcat Version: *7.0.40 *Session Timeout: *30 minutes Hi, Two of our servers reached max heap and we had to restart to resolve the issue. In troubleshooting, we noticed Http Sessions had stopped expiring 05/12th and reached 25K over a period of 11 days. I noticed the same behavior in both servers, the session expiration seem to have stopped on both of them around the same time. I'm saying this because from looking at the monitoring graphs (NewRelic) I notice the http sessions starting to climb on both servers around the same time frame. I am thinking, the issue could be two folds: #1 - The session expiration stopped working -OR #2 - The sessions themselves were created with TTLs way into the future I suspected #1. I looked that tomcat 7.0.40 source code, extracted all session/manager error messages and searched for them in the logs and didn't find any hits. I was hoping to find something about "Session expiration stopped working", but didn't find any. Can you please advise how to troubleshoot this issue? I'd very much appreciate the help. thanks, Stephen
